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Abstract 

This  thesis  presents  a  comparison  based  on  identification  rate,  of  three  clustering 
techniques  applied  to  cepstral  features  for  speaker  identification.  LBG  vector  quantization 
as  developed  by  Linde,  Buzo  and  Gray;  is  used  to  provide  benchmark  performance  for  com¬ 
parison  with  Fuzzy  clustering  (based  on  the  un-supervised  fuzzy  partition-optimal  number 
of  classes,  UFP-ONC  algorithm  by  Gath  and  Geva)  and  an  Artificial  Neural  Network,  the 
Multilayer  Perceptron. 

Cepstral  features  from  the  TIMIT,  King  and  AFIT93  corpus  speaker  databases  are 
used  to  produce  speaker-identification  classifiers  using  each  of  the  clustering  algorithms. 
The  experiment  reported  evaluates  the  speaker  identification  performance  using  the  20- 
dimensional  cepstral  features  which  were  extracted  directly  from  the  databases.  The 
speaker  databases  were  taken  from  diflTerent  recording  environments,  TIMIT  is  studio  qual¬ 
ity,  AFIT93  was  recorded  in  an  office  environment  and  King  is  recorded  telephone  con¬ 
versations.  The  performance  provides  an  indication  of  merit  for  the  clustering  techniques 
for  the  range  of  typical  recording  environments.  This  thesis  demonstrates  the  application 
of  fuzzy  clustering  for  speaker  identification.  It  is  shown  that  the  UFP-ONC  algorithm 
can  achieve  identification  rates  equal  to  the  LBG  vector  quantization  system.  LBG  vector 
quantization  provides  the  best  overall  performance  of  all  three  clustering  techniques. 
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Clustering  Techniques  in  Speaker  Recognition 

I.  Introduction 

1.1  Background 

Ensuring  that  only  the  right  people  have  access  to  buildings,  financial  records  and 
computer  systems  requires  an  effective  identification  system.  Traditional  systems  using 
locks,  combinations,  passwords  and  identification  cards  can  be  compromised  by  copies, 
decoding  or  theft  of  the  access  device. 

Automatic  identification  of  people  based  on  physical  features  of  speech  patterns,  fin¬ 
gerprints,  faces,  blood  vessels  patterns  in  the  retina  irises  and  DNA  are  all  being  researched 
to  provide  better  security  for  both  the  individuaJ  and  organizations. 

Automatic  speaker  recognition  determines  the  identity  of  a  person  from  a  known 
population  of  speakers  [38]  .  In  this  work  Cepstral  [37]  features  provide  the  patterns  which 
are  matched  to  speaker  dependent  templates. 

Potential  applications  of  speaker  identification/verification  systems  include 

•  Building  access  systems, 

•  Secure  access  to  computer  records, 

•  Covert  detection  of  criminal  activity  on  telephone  systems, 

•  Verification  of  instructions  over  communications  systems,  for  both  military  and  com¬ 
mercial  applications. 

•  Automatic  message  routing 

1.2  Problem 

This  work  examines  the  use  of  clustering  techniques  using  cepstral  coefficients  for 
speaker  identification.  A  comparison  between  vector  quantization,  fuzzy  clustering  and  an 
artificial  neural  network  (ANN)  is  made. 


The  speech  data  utilized  are  a  combination  of  phonetically  balanced  and  text-independent 
sentences  of  various  lengths.  Three  databases  provides  different  background  environments 
from  pristine  [TIMIT],  computer  room  [AFIT  corpus  93]  [11]  and  telephone  [King].  The 
AFIT  corpus  was  initiated  in  1992  by  Colombi  [ll]  for  speaker  identification,  AFIT93 
corpus  was  recorded  by  the  author  during  this  thesis  research. 

1.3  Scope 

This  research  will  implement  the  Unsupervised  Fuzzy  Partition-Optimal  Number  of 
Classes,  clustering  algorithm  [19],  and  evaluate  its  performance  for  speaker  identification. 
The  UFP-ONC  will  be  compared  against  vector  quantization  (VQ)[23]  and  the  Multilayer 
Perceptron  using  three  speech  databases.  The  three  databases  are  the  King  [25],  TIMIT 
[36],  and  an  AFIT  corpus  database  collected  during  this  research. 

1.4  Assumptions 

It  is  assumed  that  cepstral  processing  of  speech  provides  the  necessary  features  to 
uniquely  define  a  speaker.  Throughout  this  work  the  speech  recordings  are  assumed  to 
be  of  finite  length,  in  English,  and  contain  only  one  speaker.  Any  noise  inherent  in  the 
databases  will  be  considered  typical  and  no  noise  reduction  techniques  will  be  applied. 


1.5  Thesis  Organization 

Chapter  II  is  a  review  of  relevant  literature  relating  to  Fuzzy  clustering,  vector 
quantization,  neural  networks  and  speech  processing.  Chapter  III  presents  the  method¬ 
ology  used  and  the  experiments  conducted.  Chapter  IV  presents  the  results  for  each  of 
the  three  techniques,  vector  quantization,  Fuzzy  A:-Means  and  Artificial  Neural  Networks 
applied  to  speech  for  speaker  identification. 
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II.  Literature  Review 


2. 1  Introduction 

Speech  processing  brings  together  the  different  fields  of  signal  processing,  physics, 
pattern  recognition,  linguistics,  physiology,  psychology,  computer  science  and  information 
theory  [40,  42], 

In  general,  speaker  identification  methods  attempt  to  isolate  acoustic  features  which 
are  dependent  on  the  individuals  vocal  tract.  Formant  frequencies  [22],  formant  ratios 
[39]  and  pitch  contours  [4]  are  examples  of  features  which  have  been  used  in  speaker 
identification.  Automatic  classification  of  speakers,  requires  partitioning  the  feature  space 
in  a  way  that  will  separate  each  of  the  individuals. 

This  chapter  presents  a  review  of  speaker  identification,  cepstral  features,  vector 
quantization,  fuzzy  clustering  and  artificial  neural  network  literature.  For  comprehensive 
reviews  on  each  of  these  fields  refer  to  references  [6,  8,  11,  20,  23,  26,  31,  35,  38] 

2.2  Speaker  Identification 

The  task  of  speaker  identification  (SID)  is  to  find  the  absolute  identity  of  a  person 
ba.sed  only  on  their  speech.  The  speech  features  from  the  subject  are  compared  to  each 
of  the  people  in  a  known  population.  O’Shaughnessy  [38]  provides  a  review  of  speaker 
identification  and  verification.  Speaker  Verification  is  considered  the  easier  of  the  two 
problems  [38,  40].  The  subject  makes  a  claim  that  he  is  speaker  X,  and  the  verification 
system  makes  a  binary  decision,  yes  or  no  ,  by  comparing  the  subject’s  speech  features 
with  the  templates  of  speaker  X.  Speaker  identification  requires  comparison  with  all  known 
speakers  and  selects  the  best  match.  O’Shaughnessy  discusses  one  of  the  inherent  problems 
in  speaker  identification,  the  larger  the  population  of  known  speakers  the  more  difficult  the 
process  becomes.  Assuming  equal  a  priori  of  speakers,  the  probability  that  a  subject 
is  known  is  f**  =  1/N,  where  N  is  the  number  of  speakers  in  the  database.  Jayant  [35] 
discusses  the  computational  cost  of  speaker  identification,  and  notes  that  a  simple  system 
must  perform  N  decisions  to  decide  that  the  subject  is  a  known  speaker.  More  efficient 
search  algorithms  can  reduce  this  by  grouping  speakers  into  categories  such  as  male/female. 
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thus  narrowing  the  search  area.  In  “Statistical  Techniques  for  Talker  Identification”  [9] 
Bricker  et  al.  provide  insight  into  the  difficulties  presented  by  large  databases  of  speakers. 
When  the  speaker  database  is  relatively  small,  perhaps  less  than  50,  a  codebook  for  each 
speaker  can  be  kept.  However,  when  the  number  of  speakers  in  the  database  becomes 
large  there  is  immense  overhead  in  storage  and  search  times.  To  overcome  this  the  authors 
demonstrate  the  use  of  a  nearest  neighbor  classifier. 

Applications  of  SID  are  numerous,  such  as  controlling  access  to  buildings,  controlling 
access  to  privileged  information  on  computer  files  or  by  telephone.  Several  authors  have 
presented  papers  on  the  applications  for  speaker  identification,  for  a  recent  cross-section 
refer  to  [16,  35,  38,  45,  49,  50]  . 


2.3  Cepstral  Features 


The  cepstrum  (or  power  cepstrum)  provides  a  method  of  separating  the  vocal  tract 
spectrum  from  the  spectrum  of  the  vocal  tract  excitation.  Using  the  cepstrum  of  speech 
we  attempt  to  characterize  an  individual  by  their  long  term  vocal  tract  spectrum.  The 
definition  of  cepstrum  is  shown  below.  The  Fourier  Transform  of  a  time  function  is  denoted 
by 

y(a;)  =  jr{j,(0} 


The  power  spectrum  is 


y,(u;)  =  |y(a;)p 


The  cepstrum  being 


C(r)  =  jr{logyp(a;)} 


(1) 


The  power  spectrum  is  symmetric  and  real;  accordingly  we  can  expand  the  cepstrum  as  a 
Fourier  series 


logyp(u;)  =  £  (2) 

n=— oo 

where  c„  =  c_„  are  real  and  referred  to  as  cepstral  coefficients. 
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Consider  a  speech  signal,  y(t),  with  the  power  spectrum,  Yp{uj).  Assuming  the  vocal 
tract  to  be  a  linear  system,  we  can  separate  the  excitation  spectrum  X{u}),  and  the  vocal 
tract  spectrum  H{u})  as  shown  below.  The  assumption  of  stationarity  holds  for  short 
periods,  typically  40  milliseconds  [42]. 

Figure  1  shows  each  of  these  steps  applied  to  a  segment  of  the  vowel  lY,  as  shown 
in  the  upper  left  corner.  The  signal  is  256  samples,  sampled  at  8000  samples  per  second. 
The  magnitude  of  the  Fourier  transform  is  displayed  in  the  upper  right.  Note  the  large 
amplitude  components  below  200Hz,  the  glottal  pitch  and  first  formant  are  very  distinct 
approximately  85Hz  and  125Hz  respectively.  The  higher  frequency  formants  can  be  seen  in 
the  interval  between  2000Hz  and  4000Hz.  It  is  difficult  to  separate  the  excitation,  glottal 
pulse  frequency,  from  the  vocal  tract  spectrum  using  the  Fourier  transform  alone.  The 
lower  right  of  Figure  1  shows  the  logarithm  of  the  Fourier  transform,  or  power  spectrum. 
More  detail  of  the  spectrum  is  now  evident,  although  the  glottal  frequency  still  dominates 
the  lower  frequencies.  The  lower  left  plot  displays  the  cepstrum  of  the  speech,  the  glottal 
pitch  is  now  clearly  evident  at  approximately  11.5  (quefrency).  The  vocal  tract  information 
is  located  in  the  interval  between  zero  and  five.  In  a  discussion  of  feature  extraction  for 
speech  recognition  (where  identity  is  a  secondary  concern  )  Rabiner  and  Juang  discuss 
liftering,  or  normalizing  the  cepstrum  to  remove  these  low  cepstral  coefficients  which  are, 
“due  to  variations  in  transmission,  speaker  characteristics,  vocal  effects  ...  ”,  [42],  this 
is  the  information  speaker-identification  systems  seek  to  exploit. 

Consider  the  power  spectrum  signal 

YpCw)  =  X{u;)H{u;) 

Where  H{u;)  is  the  vocal  tract  transfer  function  and  A(u;)  is  the  excitation  function,  taking 
the  logarithm 

log  [YpCo;)]  =  log  [X (w)]  4-  log  [H (w)] 
followed  by  the  Fourier  Transform 

.r{log[yp(u;)]}  =  .^{log[X(u;)]}-hJF{log[H(u;)]}  (3) 
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Figure  1.  Vowel  signal,  spectrum  and  cepstrum 


we  obtain  the  cepstrum,  which  is  comprised  of  two  components,  the  excitation  and  the 
vocal  tract  transfer  function.  As  shown  in  Figure  1,  the  cepstrum  is  very  effective  at 
separating  the  pitch  from  the  vocal  tract  spectra  when  using  voiced  speech.  Estimation 
of  the  vocal  tract  transfer  function  is  more  difficult  during  unvoiced  speech.  Unvoiced 
excitation  is  then  modelled  as  a  pseudo  random  process.  Parsons  [40:pp.l20]  discusses 
some  models  of  noise  processes  associated  with  different  acoustic  phonetics. 

Cepslral  coefficients  can  be  obtained  from  both  the  Fourier  transform  (FFT)  and 
Linear  Predictive  coding  (LPC).  Furui  compared  both  features  for  speaker  identification 
[18]  and  concluded  both  achieved  the  same  performance,  although  LPC-cepstrura  is  much 
faster  to  compute. 

As  the  cepstral  coefficients  used  in  this  thesis  are  derived  from  the  LPC-cepstrum,  a 
brief  explanation  of  Linear  predictive  coding,  the  use  of  an  all-pole  model  and  the  LPC- 
cepstrum  follows.  It  is  generally  accepted  that  an  all-pole  model  of  the  vocal  tract  is  used 
in  linear  predictive  coding  [40],  because  of  its  simplicity,  and  the  ability  to  model  zeroes 
introduced  by  the  nasal  tract.  Parsons  [40]  discusses  the  conditions  for  approximating  a 
zero  by  a  number  of  poles.  Increasing  the  order  of  the  all-pole  model  provides  an  effective 
and  tractable  model  of  the  human  vocal  tract.  The  derivation  of  the  LPC-cepstrum  is 
based  on  this  all-pole  model.  The  aim  of  linear  prediction  is  to  estimate  the  output  of  a 
linear  time-invariant  system  based  on  past  values  of  input  and  output. 

«  p 

j/[7i]  =  '^b\j]x[n-  j]  -  (4) 

j=0  1=1 

>  v<  '  ^  ^ 

current  and  previous  inputs  previous  outputs 

Where  y[n]  is  the  predicted  output,  a[i]  and  6  [7]  are  the  predictor  coefficients.  When  y 
converges  we  have 

p  1 

^o[i]j/[n-t]  =  ^6[;]i[n-j] 

1=0  j=0 

Taking  the  z-transform 


p 


F(2)^o[i] 


1=0 


;=0 
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Taking  the  ratio  of  outpu  t,  to  input 


tsO 


This  is  the  vocal  tract  spectrum,  H{z) 


E*b'l 


.~3 


j-0 


Hiz)  = 

S  a  [i] 

1=0 

By  using  an  all  pole  model  the  numerator  reduces  to  a  gain  term 

Hiz)  = 


•=o 

The  output  is  now  a  function  of  previous  outputs  only,  and  setting  a  [0]  =  1 

H{z)  = - - 


(5) 


1 

»=1 

Where  we  redefine  a  [i]  to  be  —a  [i]  for  convenience.  This  is  usually  represented  as 


H{z) 


A{z) 


p 

Where  A{z)  =  To  compute  the  cepstral  coefficients  from  the  LPC  model  the 

i=0 

logarithm  of  the  transfer  function  is  taken 


log  [<tIA{z)\  =  logtr  +  ^  c„z  " 

n=:l 
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differentiating  with  respect  to  z“*,  and  making  a  Taylor  series  expansion 

n— I 

C„  =  -a„  -  ^  X!  for  n  >  0  (6) 

*=1 

where  oq  =  1,  and  a*  =  0  for  ^  >  p. 

The  log  power  spectrum  becomes 

log[aV|A(e^“)n  =  E  (7) 

n=— oo 

where  ,  c„  =  c_„,  are  the  LPC-cepstral  coefficients. 

The  cepstral  features  of  a  vowel  lY  are  shown  in  Figure  1,  the  arrows  indicate  the 
glottal  pitch  and  the  first  three  formants.  Formants  are  the  resonant  frequencies  of  the 
vocal  tract  due  to  the  glottal  pulses  [40].  The  cepstral  features  can  be  used  to  track  the 
pitch  and  to  characterize  the  vocal  tract  transfer  function  of  a  speaker  [38,  40). 

Figure  2  shows  the  cepstral  features  of  two  speakers  from  King  .  The  figure  shows 
the  feature  vectors  of  voiced  speech  during  a  60  second  conversation.  While  the  text  of  the 
speech  is  not  identical,  there  are  still  visible  differences  between  the  speakers.  The  purpose 
of  applying  clustering  algorithms  to  these  features  is  to  characterize  each  speaker  by  the 
uniqueness  of  the  patterns  evident  in  Figure  2 

2.4  Vector  Quantization 

Vector  quantization  is  procedure  for  representing  a  signal  by  a  number  of  symbols  or 
codewords.  Vector  quantization  is  widely  used  in  communication  systems,  where  an  analog 
input  is  transmitted  as  a  sequence  of  binary  codes.  Reconstruction  of  the  signal  is  achieved 
using  the  codebook  in  reverse,  mapping  the  codewords  into  an  analog  signal.  The  objective 
of  vector  quantization  is  to  determine  the  optimal  codebook  that  ensures  reconstruction 
with  minimal  distortion.  Gersho  and  Cuperman  [20]  discuss  the  applications  of  VQ  to 
speech  transmission  including  different  codebook  design  methods.  There  are  a  number  of 
ways  to  design  VQ  codebooks,  in  this  research  the  technique  developed  by  Linde,  Buzo 
and  Gray  [30]  is  used.  For  an  excellent  review  on  vector  quantization  and  codebook  design 
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NngapealBf  1 


toaure 

King  speaker  2 


20 

feature 


Figure  2.  Cepstral  feature  vectors  from  King  speakers  One  and  Two.  (60  seconds  of 
conversation  from  Session  one) 


the  reader  should  refer  to  the  1984  paper  by  Gray  [23].  The  Linde,  Buzo  and  Gray  (LBG) 
technique  forms  a  codebook  by  progressive  splitting  of  codewords.  The  LBG  algorithm  is 
implemented  as  follows  : 


1.  Find  the  mean  of  the  data  set,  use  this  as  the  initial  ifc-dimensional  codeword,  j/o. 

2.  Double  the  size  of  the  codebook  by  splitting  each  existing  codeword 

•  Vn  =  J/n(l  +  c), 

•  yn=  yn(l  -  (), 

•  where,  n,  ranges  from  1  to  the  current  size  of  the  codebook, 

•  (0.01  <  €  <  0.05)  is  a  scalar  multiplier. 

3.  Iteratively  recalculate  the  centroids,  tising  A:-means,  to  obtain  the  optimal  centroid 
locations. 

4.  Repeat  steps  two  and  three  until  the  codebook  is  the  desired  size. 
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Figure  3.  Vector  Quantization  codebook  design  algorithm  [42] 

A  flow  chart  of  the  algorithm  is  shown  in  Figure  3.  The  Classify  Vectors  is  the 
nearest-neighbor  algorithm  [17]  (k-nn)  and  the  Find  Centroids  is  the  l:-means  algorithm 
[47]. 

A  simple  example  of  codebook  splitting  using  a  2-dimensional  data  set  is  illustrated  in 
Figure  4.  Figure  4(a)  shows  an  initial  codeword,  the  mean.  The  codebook  is  split  two,  the 
LBG  algorithm  computed  and  the  two  new  codewords  are  shown  in  3(b).  A  final  splitting 
produces  four  codewords  which  represent  the  four  classes.  In  order  to  terminate  with  these 
four  codewords  the  user  must  know  that  only  four  centers  are  present.  Otherwise  the 
algorithm  will  continue  increasing  the  codebook  size,  which  may  not  be  beneficial. 

Vector  Quantizer  codebooks  provide  two  outputs  for  each  input  vector.  These  out¬ 
puts  are  the  location  number  of  the  closest  codeword,  and  the  distortion  between  the  input 
vector  and  the  codeword.  The  distortion  figure  is  used  for  speaker  identification.  The  un¬ 
known  speaker’s  identity  is  decided  as  the  identity  of  the  speaker  whose  codebook  returns 
the  lowest  distortion.  Figure  5  shows  an  overview  of  a  vector  quantizer  based  classifier. 
For  a  ^-dimensional  input  vector,  x,  and  the  nearest  codeword,  i,  the  Euclidean  distortion 
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Figure  4.  (a)  Initial  codeword  (b)  First  split  (c)  Final  Codewords 


Figure  5.  Vector  quantizer  based  classifier 


is 


d(i,x)  =  ||x  -  x|p  (8) 

<=o 

In  order  to  identify  a  speaker  the  distortion  of  the  utterance  is  calculated  using  each  of  the 
known  speakers  codebooks.  The  identity  is  chosen  using  the  lowest  distortion. 

2.5  Fuzzy  Clustering 

Fuzzy  clustering  techniques  provide  an  intuitive  and  useful  tool  in  pattern  recogni¬ 
tion.  Bezdek  [6]  provides  a  very  comprehensive  discussion  of  fuzzy  algorithms  and  their 
application  to  pattern  recognition.  Fuzzy  mathematics  assigns  memberships  to  each  data 
point,  one  membership  for  each  of  the  cluster  centers  in  the  data  space.  A  sample  which  is 
close  to  a  cluster  center,  or  centroid,  has  a  high  membership,  close  to  one,  in  that  cluster 
and  much  lower  memberships  in  the  other  clusters.  The  degree  of  membership  provides  an 
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indication  of  how  typical  the  sample  is  in  a  given  cluster.  Consider  a  sample  due  to  a  noise 
process,  which  is  an  outlier  in  a  data  set.  Conventional,  or  hard,  clustering  algorithms 
will  assign  the  point  to  a  cluster  even  if  it  is  detrimental  to  do  so.  In  comparison,  fuzzy 
clustering  will  assign  the  data  point  very  low  memberships  in  all  clusters,  indicating  that 
it  is  not  typical  of  the  data  set.  The  membership  value  Uij,  of  a  sample  j,  in  cluster  i  is 
defined  in  the  interval 

0  <  Uij  <  1 

The  membership  value  is  generally  based  on  the  inverse  of  the  distance  from  the  cluster 
center  to  the  sample  point.  The  membership  is  defined  as  follows  : 


EiMJf/.n) 

t=i 


(9) 


Uij  :  is  the  membership  of  sample  Xj  in  cluster  Vi 
Where  ;  A'  :  is  the  total  number  of  clusters. 

d{Xj,V]i.)  :  is  the  distance  from  sample  Xj  to  centroid  cluster  Vi 
Figure  6  illustrates  the  membership  function  of  a  point  as  a  function  of  normalized 
distance  from  a  cluster  center.  The  figure  shows  a  typical  membership  function,  however 
these  vary  between  authors  and  applications  [6,  10,  19,  29]. 

Fuzzy  logic  came  into  prominence  with  the  1965  paper  “Fuzzy  Sets”  by  Lotfi  Zadeh 
[51].  Since  then  fuzzy  theory  has  been  applied  to  ir»any  areas  including  pattern  recognition. 
Ruspini  provided  one  of  the  first  applications  of  fuzzy  logic  in  clustering,  he  extended  the 
conventional  A;-means  algorithm  into  the  Fuzzy  fc-Means  (FKM)[44].  Figure  7  is  an  example 
Ruspini  used  to  illustrate  the  memberships  of  data  points  using  fuzzy  clustering.  The  top 
of  the  figure  shows  two  triangular  shaped  clusters,  commonly  known  as  Ruspini’s  butterfly. 
The  lower  section  of  the  figure  shows  the  memberships  in  each  of  the  two  classes. 

a:-direction  The  work  by  Ruspini  was  extended  by  Dunn  [13]  with  the  proof  of  con¬ 
vergence  for  the  Fuzzy  c-Means  published  in  1980  and  1987  [5,  7]. 
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The  Fuzzy  AMueans  clustering  algorithm  is  described  in  pseudo-code  below  : 


1.  Select  the  number  of  clusters  required  and  initialize  their  positions  randomly, 

2.  Compute  the  membership  of  each  data  point  for  all  the  clusters, 

3.  Compute  new  cluster  centers  using  the  n««  membership  values, 

4.  Compute  the  membership  of  each  data  point  for  all  clusters, 

5.  Is  the  m^[|uij  -  «ij  |]  less  than  the  stopping  criteria? 

•  IF  yes,  THEN  stop. 

•  ELSE  repeat  steps  3  to  5. 

6.  Save  final  cluster  positions  for  use  in  classification. 

Where  m^[|ti';j  —  is  the  objective  function  and  u  is  the  new  membership  matrix. 


2.5.1  The  Fuzzy  k-Means  Algorithm.  The  fuzzy  it-means  algorithms  is  shown 
below.  The  FKM  is  listed  in  C  code  and  MATLAB  script  in  the  Appendix. 


1.  Initialize  the  centroids,  Vi,  using  any  suitable  method, 

2.  Compute  the  distance  from  each  centroid  to  each  data  sample  using 

d{Xj,Vi)  =  {X,-VifA-^(X,-V.)  (10) 


where  A  is  a  positive  definite  matrix.  If  A  is  the  identity  matrix  then  the  distance  measure  is  the 
Euclidean  distance. 


3. 


Compute  the  next  iteration  of  centroids  V,  for  some  q  >  0. 
fuzziness  and  in  this  work  q  =  2. 

N 

V  _ 

V,  -  — 

j=) 


The  parameter,  q,  is  known  as  the 


4.  Save  the  previous  memberships, 

5.  Update  the  data  memberships,  u'l,  using  Ekiuation  10 


[1MA„V0](^-^) 

K 

*=1 


6.  Calculate  the  objective  function 

max[|«<j  -  u,j|] 

•} 

7.  If  the  (objective  function  <  e),  then  STOP  otherwise  GOTO  step  3  and  continue  iterating. Where 

f  €  [0,1]  . 


Ifi 


The  fuzzy  Ar-means  and  the  hard  k-means  both  form  spherical  clusters  around  the 
centroids,  this  does  not  always  reflect  the  true  shape  of  clusters.  To  better  exploit  clus¬ 
ter  shapes  Gustafson  and  Kessel  [24]  introduced  the  Fuzzy  covariance  matrix  into  the 
FKM  algorithm.  This  work  was  extended  by  Gath  and  Geva  in  their  Unsupervised  Fuzzy 
Partition- Optimal  Number  of  Clusters  (UFP-ONC)  algorithm  [19].  In  their  paper  Gath 
and  Geva  introduce  a  method  for  determining  the  optimal  number  of  cluster  present  in  a 
data  set.  Optimality  is  determined  using  cluster  validity  criteria.  The  cluster  validity  mea¬ 
sures  are  the  fuzzy  hyper-volume,  partition  density  and  average  partition  density.  Simply 
put,  each  cluster  is  specified  by  a  centroid,  a  fuzzy-covariance  matrix  and  the  a  priori  prob¬ 
ability  of  the  cluster.  The  fuzzy  covariance  specifies  the  local  geometry  of  each  cluster,  this 
technique  is  widely  used  in  image  processing  for  the  detection  of  lines  for  applications  such 
as  computer  vision  and  satellite  image  processing.  Cluster  validity  measures  assess  each 
cluster  by  measuring  its  volume  (in  n-dimensions)  based  on  the  fuzzy  covariance  matrix. 
This  can  be  thought  of  as  building  a  shell  around  the  cluster,  with  the  shell  wall  being 
one  standard  deviation  from  the  center  in  each  of  the  dimensions.  The  smaller  this  volume 
the  more  compact  the  cluster,  which  is  a  desirable  outcome.  However,  in  the  extreme  case 
where  hyper- volume  is  maximized  each  data  point  could  be  considered  a  cluster.  The  ben¬ 
efit  of  the  clustering  would  be  lost,  so  a  second  criteria,  the  cluster  density,  is  introduced 
to  ensure  that  the  clusters  are  both  compact  and  densely  populated. 

To  determine  the  optimal  number  classes  the  Unsupervised  Fuzzy  Partition- Optimal 
Number  of  Classes  (UFP-ONC)  algorithm  computes  the  cluster  validity  criteria  starting 
with  two  centroids.  The  number  of  centroids  is  then  increased  to  three,  again  the  cluster 
validity  measures,  fuzzy  hyper- volume,  partition  density  and  average  partition  density  are 
calculated.  The  number  of  centroids  is  progressively  increased  until  the  maximum  (set 
by  the  user)  is  reached.  By  examining  the  cluster  validity  measures  against  the  number 
of  clusters  an  optimum  may  be  found.  This  optimum  should  provide  small  fuzzy  hyper- 
volume,  high  partition  density  and  high  average  partition  density.  The  FMLE  section 
of  the  UFP-ONC  must  be  applied  carefully,  if  the  UFP-ONC  is  initialized  poorly,  the 
algorithm  will  not  converge  to  the  true  cluster  centers.  This  is  due  to  the  fact  that  the 
distance  measure,  Equation  11,  and  the  fuzzy  covariance  matrix,  confine  the  centroids  to 
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a  small  area,  effectively  restricting  the  search  of  the  feature  space.  To  overcome  this  the 
UFP'ONC  uses  the  fuzzy  Ir-means  algorithm  to  initialize  the  centroids  and  then  refines  the 
centroid  locations  using  the  fuzzy  maximum  likelihood  estimation  algorithm. 

The  UFP-ONC  algorithm  in  pseudo-code  can  be  summarized  as 

1.  Set  the  number  of  clusters  at  two 

2.  Repeat  until  maximum  number  of  clusters  computed 

•  Compute  the  centroid  locations  using  the  fuzzy  fc-means 

•  Compute  the  Fuzzy  Maximum  Likelihood  Estimation  algorithm 

•  Compute  the  cluster  validity  measures, 

3.  Analyze  the  validity  measures  to  determine  the  optimal  number  of  classes 
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2.5.2  The  Fuzzy  Maximum  Likelihood  Estimation  Algorithm.  The  fuzzy  maxi¬ 
mum  likelihood  estimation  (FMLE)  [19]  algorithm  is  shown  below.  The  Ccode  is  included 
in  the  Appendix. 


1.  Initialize  the  centroids  using  the  fuzzy  h-means 

2.  Set  the  initial  data  posterior  (memberships  in  the  FKM)  to 

/((ilJVj)  =  where  K  is  the  number  of  clusters 
K 

3.  Set  the  initial  fuzzy  covariance  to  the  identity  matrix 

4.  Compute  the  a  priori  probability  of  each  cluster 

>=i 


5.  Compute  the  exponential  or  fuzzy  Mahalanobis  distance  from  each  centroid  to  each  data  sample 
using 

dl{X„V,)  =  [{Xj  -  VifF-^X,  -  V;)/2]  (11) 

where  Fi  the  fuzzy  covariance  matrix  is  defined  by  Ek]uation  12. 

6.  Compute  the  next  iteration  of  centroids  V, 


J2^inx,)x, 


J=1 


7,  Save  the  previous  a  posterior  probabilities  (memberships), 


him 


8.  Update  the  data  memberships,  h(i|Xj)  using 


9.  Calculate  the  objective  function 

niM  [|A(fjjri)  -  A(»iA'j)|] 


10.  If  the  (objective  function  <  «),  then  STOP  otherwise  continue  iterating  using  Equation  12  below 

11.  Compute  the  fuzzy  covariance  matrix,  Fi  for  each  centroid 


Fi  = 


N 

53fc(.jx>)(x,-v.)(^,-v;)’ 

y=i _ 

N 


(12) 


j=i 
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Fuzzy  techniques  have  been  applied  in  conjunction  with  vector  quantization  for  use 
in  a  speech  recognition  system.  Tseng,  Sabin  and  Lee  [48]  used  a  fuzzy  vector  quantizer 
(FVQ)  to  reduce  the  amount  of  training  data  required  for  a  Hidden  Markov  Model  (HMM). 
In  that  application  an  LBG  based  vector  quantizer  was  modified  to  output  a  membersliip 
function  vector.  This  combination  of  LBG  and  fuzzy  membersliips  provided  a  technique 
for  reducing  the  distortion  in  the  output  of  the  vector  quantizer.  The  memberships  were 
used  to  assist  the  Hidden  Markov  model  to  determine  the  state  transition  probabilities. 
Fuzzy  techniques  and  vector  quantization  fusion  has  also  been  reported  by  Asakawa  et  al 
[2,  3].  This  work  used  the  LBG/Fuzzy  combinations  for  improving  the  fidelity  of  speech 
transmitted  using  low  bit  rates  of  2400bps.  These  applications  of  fuzzy  techniques  illustrate 
that  some  benefit  is  offered  by  introducing  fuzzy  methods  into  speech  processing  problems. 

2.6  Artificial  Neural  Networks 

Artificial  Neural  Networks  comprise  a  wide  variety  of  different  algorithms  and  archi¬ 
tectures.  They  all  share  common  attributes  of  highly  interconnected  nodes  and  a  learning 
equation.  The  success  of  the  human  neurological  system  is  the  basis  for  artificial  neural 
networks.  From  the  pattern  recognition  standpoint  ANNs  are  non-linear  classifiers.  The 
multilayer  perceptron  can  learn  arbitrarily  complex  decision  boundaries,  provided  enough 
nodes  are  used  [12].  Rogers  and  Kabrisky  [43]  discuss  the  different  types  of  non-linearities 
that  can  be  used.  Lippmann  [31]  in  his  excellent  tutorial,  provides  a  extensive  discussion 
of  the  multilayer  perceptron  (MLP)  and  the  back-propagation  algorithm  which  trains  the 
network. 

The  back-propagation  algorithm  is  a  gradient  descent  algorithm  designed  to  minimize 
the  mean  square  error  between  the  actual  output  of  the  network  and  the  desired  output. 
The  training  process  is  supervised  training,  this  means  that  the  data  set  is  labelled  to 
indicate  which  class  each  of  the  feature  vectors  represents.  Each  feature  vector  is  presented 
to  the  MLP,  each  feature  is  weighted  and  fed  forward  to  a  hidden  layer  of  nodes.  Each 
node  sums  its  input  and  then  applies  the  non-linear  (typically  sigmoid)  function.  The  first 
hidden  layer  outputs  are  then  weighted  and  fed  to  the  following  layer,  this  continues  until 
the  final  output  layer  is  reached.  The  output  vector  is  compared  to  a  desired  output  set 
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by  the  user.  The  error  is  then  back- propagated  through  each  layer  of  nodes.  At  each  layer 
the  weights  are  adjusted  using  the  back- propagation  algorithm.  Recent  research  at  AFIT 
by  McCrae,  Keller  and  Martin,  using  neural  networks  has  been  successful.  McCrae  [34] 
used  an  MLP  for  color  image  segmentation,  as  a  pre-processor  for  a  target  identification 
system.  Keller  [27]  successfully  used  neural  networks  for  personnel  identification  by  fusing 
face  data  and  speech  data  and  Martin  [32]  applied  neural  networks  to  radar  identification 
of  non-cooperative  targets. 

2. 7  Conclusion 

This  chapter  provides  a  review  of  literature  relating  to  speaker  identification,  cepstral 
features,  minimum  dist  j-tion  classification  and  clustering  techniques.  The  review  of  the 
cepstral  features  included  both  the  Fourier  Transform  method,  and  the  Linear  Predictive 
Coding  method  of  finding  the  cepstral  coefficients.  The  cepstral  features  provide  informa¬ 
tion  about  the  vocal  tract  transfer  function  of  a  speaker.  By  using  clustering  algorithms 
we  attempt  to  find  a  unique  set  of  parameters  which  uniquely  identify  that  person. 

Clustering  techniques  attempt  to  partition  the  feature  space  into  regions  defined  by 
a  centroid  and  its  neighborhood.  Having  partitioned  a  feature  space  we  can  construct 
a  classifier  which  provides  the  heart  of  the  speaker  identification  system.  The  clustering 
algorithms  reviewed  are  vector  quantization,  specifically  the  algorithm  by  Linde,  Buzo  and 
Gray,  fuzzy  clustering,  with  emphasis  on  work  by  Gath  and  Geva  and  artificial  neural 
networks.  Vector  quantization  methods  are  now  well  developed  and  widely  used  in  com¬ 
munication  system,  in  this  work  they  provide  a  baseline  for  evaluating  fuzzy  clustering  and 
ANN  performance.  Fuzzy  clustering  offers  a  technique  for  dealing  with  data  points  which 
are  near  the  boundary  of  two  partitions.  These  points  often  present  difficulties  for  cluster¬ 
ing  algorithms  as  they  may  be  from  either  cluster.  Fuzzy  clustering  attempts  to  alleviate 
this  problem  by  assigning  memberships,  the  higher  the  membership  the  more  likely  that 
the  point  is  in  a  given  cluster.  Artificial  neural  networks  provide  a  dramatically  differ¬ 
ent  approach  to  clustering,  the  network  is  able  to  construct  arbitrarily  complex  decision 
boundaries  to  partition  the  feature  space.  ANN’s  use  relatively  simple  functions  such  as 
sigmoids  inside  a  highly  interconnected  network.  All  three  methods  attempt  to  partition 
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the  feature  space  through  a  learning  process.  These  techniques  are  extremely  useful  when 
the  process  is  too  complex  to  be  deterministically  modelled. 

Chapter  III  discusses  the  structure  of  the  data  sets,  including  the  use  of  separability 
measures,  and  finally  the  implementation  of  the  clustering  algorithms. 
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III.  Approach  and  Methodology 


3.1  Introduction 

Three  different  clustering  methods  were  used  in  this  speaker  identification  experi¬ 
ment,  vector  quantization,  fuzzy  clustering  and  a  multi-layer  perceptron.  The  following 
sections  describe  the  each  of  the  data  sets,  the  speech  capture  and  preprocessing,  and  the 
implementation  of  each  of  the  clustering  algorithms. 

3.2  Data  Sets 

Three  data  sets  were  used  in  this  work,  the  TIMIT  and  King  speech  databases  and  a 
local  AFIT  corpus  which  was  recorded  during  this  research.  The  AFIT93  corpus  comprises 
twelve  speakers,  eleven  male  and  one  female.  Ten  of  the  speakers  were  recorded  in  ten 
sessions  over  a  three  week  period.  The  remaining  two  were  recorded  for  seven  sessions 
over  the  same  three  week  period.  Each  session  comprises  the  person’s  full  name,  uttered 
three  times,  followed  by  three  sentences  based  on  the  phonetically  balanced  sentences  of 
the  TIMIT  database. 

A  portion  of  the  DARPA  TIMIT  Acoustic  Phonetic  Continuous  Speech  Database  [36] 
was  used  to  provide  a  corpus  of  ten  speakers,  seven  male  and  three  female,  each  speaking 
ten  sentences.  The  TIMIT  database  provides  good  quality  recordings,  with  a  signal  to 
noise  ratio  of  36.72dB.  The  TIMIT  sentences  used  were  the 

•  SA 

•  SX 

•  SI 

Examples  of  the  phonetically  balanced  TIMIT  phrases  are  included  in  the  Appendix. 

The  King  database  was  collected  by  ITT  Aerospace  and  comprises  two  recording 
methods, wide  and  narrow  band.  The  narrowband  recordings  of  the  first  twelve  speakers 
were  used  in  this  work.  Each  speaker  is  recorded  in  ten  sessions,  with  each  session  60 
seconds  in  length.  The  King  database  was  collected  in  two  stages  using  different  equipment, 
the  first  five  sessions  are  of  much  higher  quality  than  the  second  five.  Colombi  [11]  provides 
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details  of  the  signal  to  noise  ratios  with  and  without  silence  frames,  the  maximum  is 
14.75dB,  significantly  lower  than  TIMIT. 

The  AFIT93  recordings  were  made  in  a  computer  room,  using  an  Ariel  Proport  and 
SUN  Sparc  2  station.  The  speech  was  recorded  at  16KHz  and  later  down  sampled  to  8KHz 
in  keeping  with  the  sampling  rate  of  King  and  TIMIT. 


3.3  Data  Pre-processing  and  Feature  Extraction 

Each  of  the  data  sets  was  processed  using  the  Entropic  Signal  Processing  System 
[14]  which  is  a  commercial  software  package  comprising  a  library  of  programs  for  speech 
processing.  The  data  was  pre-emphasized  with  a  1  —  0.97^“^  filter,  followed  by  a  Ham¬ 
ming  window  using  256  samples  (32ms)  per  frame,  with  consecutive  frames  overlapping  by 
85  samples.  The  next  stage  of  processing  involves  two  processes,  LPC-cepstral  coefficient 
extraction  and  calculation  of  formants.  The  formant  information  is  used  to  calculate  the 
probability  of  voicing  of  the  frame  of  speech.  The  probability  of  voicing  is  then  appended 
to  each  cepstral  feature  vector  and  used  to  select  only  feature  vectors  which  have  greater 
thaji  10%  probability  of  voicing.  These  feature  vectors  are  then  used  in  the  clustering 
experiments.  Figure  8  provides  an  overview  of  the  pre-processing  system.  The  follow¬ 
ing  code  is  C  script  file  [21]  containing  the  ESPS  commands  which  implement  the  data 
preprocessing. 

S ! /bin/csh 

* 

*  This  script  uses  ESPS  to  extract  cepstral  features  fros  the  TIMIT  corpus 
t  database  collected  by  Texas  Instrusents 

t 

foreach  FILE  (e.sd) 

filter  -f  filter  -P  pree■p_p^tra■8  transit. sd  transit. f It. sd 
refcof  -P  ../Prefcof  transit. f It. sd  transit. ref .cof 
spectrans  -■"CEP"  transit. ref .cof  transit. cep 
formant  -p.97  -w.032  -W1  -i.0106  -F  transit. sd 
pplain  -e2  transit. fO  >  pv 

addfea  -f  prob.voice  -c"Adding  prob.voice"  pv  tremsit.cep 
set  TEMP  ■  $FILE:r 

select  -q  "prob_ voice  >  .1"  -o  STEMP.cep  transit. cep 

end 

The  final  pre-processing  step  was  to  partition  the  databases  into  equal  halves  for 
training  and  testing  of  the  classifiers. 
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Table  1.  Data  Selection  used  for  Training  and  Testing 


Test  No. 

Sessions  used  in  Training  Sessions  used  in  testing 

1 

1 

1,  2,  3,  4,  5  (6,  7)* 

2 

1,2 

1,  2,  3,  4,  5  (6,  7)- 

3 

1,2,3 

1,  2,  3,  4,  5  (6,  7)* 

4 

1,2, 3,4 

1,  2,  3,  4,  5  (6,  ly 

5 

1,2,3,4,5 

1,  2,  3,  4,  5  (6,  7)* 

6* 

1,2,3,4,5,6 

1,2,  3,  4,  5,  6,7 

7* 

1,2,3,4,5,6,7 

1,  2,  3,  4,  5,  6,  7 

♦  AFIT93  only 


The  first  seven  days  of  the  AFIT93  database  was  split  into  training  and  testing  sets. 
This  provided  seven  sessions  of  data  with  each  session  comprising  a  training  and  testing 
set. 

All  ten  of  the  King  sessions,  for  speakers  one  to  speaker  twelve,  was  divided  into 
equal  training  and  testing  sets. 

TIMIT  speakers  fcmmO,  fcrhO,  fedwO,  mcmjO,  mefgO,  mhpgO,  mjlsO,  mmwhO,  mprkO, 
mrtkO,  were  used  in  the  experiment.  The  utterances  from  each  speaker  were  divided  into 
training  and  testing  sets  in  the  same  manner  as  for  King  and  AFIT93. 

The  identification  rate  was  recorded  for  classifiers  trained  using  an  increasing  number 
of  sessions.  Based  on  the  assumption  that  codebooks  designed  with  data  from  a  number  of 
sessions  would  better  represent  the  long  term  statistics  of  the  speakers  cepstral  features,  it 
is  expected  that  the  identification  rate  should  increase  with  the  number  of  training  sessions 
present.  Table  1  shows  which  training/testing  data  was  used  for  each  of  the  experiments. 

3.4  Vector  Quantization 

The  vector  quantization  speaker  identification  system  was  developed  using  the  ESPS 
libraries  in  an  identical  method  to  that  used  in  previous  work  at  AFIT  by  John  Colombi 
[11].  The  training  data  for  each  database  was  used  to  generate  a  vector  quantizer  codebook 
for  each  speaker  using  the  vqdes  (LBG)  algorithm  available  in  ESPS.  These  codebooks 
were  then  combined  into  a  global  codebook  which  was  used  as  the  classifier. 
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The  identification  process  presents  the  feature  vectors  of  an  utterance  to  the  global 
codebook,  the  utterance  is  then  compared  against  each  of  the  speaker  codebooks.  For  each 
codebook  the  distortion  is  computed  using  the  Euclidean  distance  of  the  feature  vectors 
to  each  of  the  codewords.  The  classifier  then  chooses  the  speaker  codebook  which  has  the 
lowest  distortion. 

Define  D(wj),  the  utterance  distortion 

K 

^(‘^•)  =  Xlminikt  -  (13) 

1:  =  1  ^ 

we  choose  class  Wi  if 

Diui )  =  min  D{uii) 

Ulc  :  feature  vector  to  be  classified 
Cij  :  codeword  j  from  speaker  i 
where:  K  :  is  the  number  of  feature  vectors  in  the  utterance. 

j  ;  is  the  number  of  codewords  in  each  codebook,  and 
i  :  is  the  number  of  known  speakers. 

In  this  experiment,  the  codebooks  have  64  codewords  for  each  speaker,  so  for  example 
the  AFIT93  VQ  classifier  decision  is  given  by 

K 

Diu};)—  min  min  llut  — c,j|| 

3.5  Fuzzy  Clustering 

The  identical  training  data  as  used  in  the  VQ  classifier  was  clustered  to  produce  a 
set  of  centroids  for  each  session,  for  each  speaker.  The  Fuzzy  clustering  algorithm  was 
written  by  the  author  using  ANSI  C  [28]  and  includes  routines  from  ‘Numerical  Recipes 
in  C”  [41].  The  fuzzy  clustering  algorithm  is  performed  in  two  distinct  steps.  The  initial 
clustering  is  performed  using  the  Fuzzy  A--means  (FKM).  After  the  FKM  has  converged  the 
second  pha^e  introduces  the  Fuzzy  Maximum  Likelihood  Estimation  (FMLE)  algorithm 
published  by  Gath  and  Geva.  The  FMLE  algorithm  computes  a  covariance  matrix  for  each 
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of  the  centroids.  This  is  then  used  in  the  calculation  of  the  distances  of  data  points  from 
the  centroid.  The  distances  are  subsequently  used  for  membership  calculations.  The 
UFP'ONC  algorithm  is  discussed  in  detail  in  chapter  II. 

A  number  of  issues  were  encountered  during  the  application  of  the  UFP-ONC  to 
speaker  identification.  The  calculation  of  the  inverse  of  fuzzy  covariance  matrices  posed 
significant  difficulties  that  were  not  anticipated.  The  fuzzy  covariance  matrix  was  of¬ 
ten  singular  or  near  singular,  which  prevents  finding  the  inverse.  Initially  the  inverse 
was  calculated  using  LU  decomposition,  however  this  was  replaced  with  Singular  Value 
Decomposition  (SVD).  SVD  provides  a  means  of  detecting  singularity  and  was  used  to 
compute  the  pseudo-inverse  of  the  covariance  matrix  when  the  complete  inverse  could  not 
be  computed.  There  appear  to  be  a  number  )f  reasons  for  the  covariance  matrix  becoming 
singular.  The  most  obvious  case  is  where  points  are  coUinear  or  coplanar.  The  most  likely 
source  of  problems  is  the  exponential  (Equation  11)  distance  measure.  The  distances  can 
become  extremely  large  and  cause  the  memberships  become  very  small,  often  zero.  This, 
in  turn,  is  reflected  in  the  calculation  of  the  fuzzy  covariance  matrix,  which  may  lead  to  it 
becoming  singular  even  though  the  members  of  the  cluster  are  not  coplanar. 

Another  issue  which  was  encountered  is  the  problem  of  repeated  centroids.  This 
occurs  when  two,  or  more,  centroids  are  located  at  exactly  the  same  point.  The  repeated 
centroid  problem  is  mainly  attributed  to  the  fuzzy  F-means  algorithm  which  is  used  to 
make  the  initial  estimate  of  the  centroids.  This  problem  is  not  confined  to  fuzzy  algorithms. 
Bezdek  [6]  discusses  these  types  of  problems  and  a  number  of  techniques  to  prevent  this. 
A  solution  to  this  problem  was  not  found,  although  it  is  an  important  addition  that  thv 
UFP-ONC  requires  to  ensure  its  correct  operation. 

These  issues,  which  the  author  considers  to  be  the  fragility  of  the  UFP-ONC,  require 
additional  research. 

The  programs  were  run  using  SUN  Sparc  2,  Sparc  10,  IBM  RS6000  and  Silicon 
Graphics  Iris  4D  computers.  A  full  listing  of  the  code  is  included  in  the  appendix.  The 
C  program  reads  a  configuration  file  which  details  the  parameters  to  be  used  for  a  given 
clustering  run.  An  example  of  the  configuration  file  is  also  included  in  the  appendix. 
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S.6  Fuzzy  Classifier 

The  final  component  of  the  speaker  identification  is  the  classifier.  Gath  and  Geva 
achieved  classification  by  examining  the  membership  values  after  the  clustering  process  had 
converged.  A  few  comments  about  the  fuzzy  majcimum  likelihood  estimation  algorithm  are 
required  as  it  has  considerable  impact  on  the  design  of  the  classifier. 

The  FMLE  uses  fuzzy  covariance  matrices  to  define  local  neighborhoods  around  each 
centroid,  this  combined  with  the  distance  measure  produces  memberships.  In  this  author’s 
experience,  these  memberships  are  very  close  to  one,  or  very  close  to  zero.  When  the  entire 
data  set  is  clustered  at  one  time  these  memberships  provide  an  excellent  classification  as 
Gath  and  Geva  [19]  demonstrated  with  Anderson’s  [1]  Iris  data. 

The  speaker  identification  experiments  conducted  in  this  work  treated  each  speaker 
separately.  The  membership  values  indicate  which  cluster  of  speaker  X  a  feature  vector 
should  be  assigned  to.  However  to  identify  a  speaker  requires  a  method  of  determining 
which  speaker  is  the  closest  using  some  form  of  distance  metric.  To  achieve  this  type  of 
distortion  ba.sed  classifica  tion,  a  rudimentary  classifier  was  designed  based  on  the  Maximum 
Likelihood  classifiers  discussed  in  Tou  and  Gonzalez  [47;Sect  4.4).  It  is  effectively  the 
same  process  as  the  LBG  classifier  shown  in  Equation  13.  The  fuzzy  maximum  likelihood 
classification  algorithm  proceeds  as  follows 

1.  Present  an  unknown  utterance 

2.  For  each  known  speaker 

•  Load  centroids  and  Fuzzy  covariance  matrices 

•  Compute  the  Fuzzy  Mahala'  obis  distances  using  Equation  11  for  all  feature 
vectors 

•  Determine  the  distance  from  the  centroid  with  the  highest  membership 

•  Total  the  log  of  these  closest  distances 

3.  Find  the  minimum  total  distance,  classify  the  utterance  to  be  from  that  speaker 
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The  classifier  was  written  in  ANSI  C  &nd  is  included  in  the  appendix.  This  classifier 
is  a  “quick  and  dirty”  approach  which  needs  closer  examination  and  some  refinement. 
However,  the  classification  system  performed  quite  well. 

5.7  Artificial  Neural  Network  Multi-layer  Perceptron 

AFIT  has  conducted  considerable  research  using  artificial  neural  networks.  The 
Multi-layer  perceptron  code  used  in  this  research  was  written  by  Curtis  Martin  for  his 
masters  thesis  [32].  The  neural  network  code  is  written  in  ANSI  C  and  was  executed 
on  the  same  computers  noted  in  the  Section  3.5  on  fuzzy  clustering.  The  code  has  been 
modified  by  this  author  to  include  the  ability  to  save  and  load  weights  and  to  remove  the 
multiple  testing  mode. 

There  are  no  hard  and  fast  methods  to  determine  how  many  hidden  nodes,  should 
be  used  in  a  neural  network.  The  basic  rules  are  too  few  and  the  error  rate  will  be  high, 
too  many  and  the  network  will  memorize  the  training  data  and  not  generalize.  Hush  and 
Horns  [26]  discuss  this  issue  at  some  length.  The  method  used  in  this  research  is  based  on 
Widrow’s  heuristic  ; 

10  *  [(m  ■\-l)*Hn  +  {H„  -f  1)  *  Co]  <  K  (14) 

m  :  is  the  dimension  of  the  feature  vectors 
H„  :  is  the  number  of  hidden  nodes 

Where 

Co  :  is  the  number  of  output  nodes  (classes) 

K  :  is  the  total  number  of  data  training  samples 

For  example  consider  designing  an  MLP  for  the  following  classification  parameters, 

•  m  =  20  dimensional  speech  features 

•  Co  =  12  speakers 

•  K  =  20,000  sample  feature  vectors 


10  ♦  [(20  -I-  1)  *  -b  (ifn  +  1)  *  12]  <  20, 000 
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=»  33  ♦  +  12  <  2000 


=>  H„  <  (2000  -  12)/33 
=>  H„  <  60.24 

S.S  Data  Separability 

Pattern  recognition  is  the  process  of  identifying  all  the  different  classes  in  a  popula¬ 
tion,  based  on  the  features  used  to  represent  the  population. 

Consider  the  data  points  in  Figure  9  below,  there  are  three  classes  of  artificially 
generated  gaussian  distributed  data.  The  {x,y)  coordinate  pairs  provide  a  feature  set 
which  can  be  used  to  separate  the  data  into  the  three  classes.  Using  this  feature  set  we 
quickly  determine  that  two  classes  overlap,  while  the  third  class  is  clearly  separated  from 
the  other  two. 


TMte  data,  3  gauaaiana  ctasaaa 


Figure  9.  Test6  data 

When  the  dimension  of  the  feature  vectors  is  increased  beyond  three  we  humans 
can  no  longer  provide  a  simple  graphical  representation,  and  the  intuitive  feel  for  class 
separability  is  lost.  The  feature  vectors  used  in  this  work  are  20-dimensional  cepstral 
coefficients,  with  the  number  of  classes  equal  to  the  number  of  speakers,  twelve  for  AFIT93 
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and  King,  and  ten  for  TIMIT.  Estimating  the  degree  of  separability  of  the  classes  indicates 
how  difficult  the  class  identification  task  is.  Additionally,  separability  measures  provide  a 
guide  to  which  pattern  recognition  algorithms  are  likely  to  be  successful.  This  aspect  of 
data  pre-processing  is  relevant  to  any  pattern  recognition  task,  such  as  image  segmentation 
of  synthetic  aperture  radar  data,  automatic  target  identification  systems  and  automated 
hand  written  chajacter  recognition.  This  remainder  of  this  section  describes  a  separability 
measure  developed  by  Fukunaga  [17],  then  applies  the  separability  measure  to  two  simple 
data  sets,  Test6  and  Anderson’s  Iris  data  [1].  The  J4  separability  measures  for  the  speech 
databases  is  presented  in  Chapter  IV. 

The  Test6  set  is  three  classes  of  2-dimensional  gaussian  distributed  data,  each  class 
containing  100  samples.  Anderson’s  [1]  Iris  data  is  4-dimensional  and  represents  the  sepal 
length,  sepal  width,  petal  length  and  petal  width  of  three  families  of  Iris.  There  are  50 
feature  vectors  from  each  of  the  Iris  families.  Iris  Sestosa,  Iris  Versicolor  and  Iris  Virginica. 
This  data  was  used  by  Fisher  in  his  development  of  linear  discriminant  analysis  [15]. 

Fukunaga  develops  a  number  of  separability  measures  of  his  book,  “Introduction  to 
Statistical  Pattern  Recognition”, [17].  The  measures  labelled  Jl,  J2,  JS  and  J4  [17:Sect 
9.2],  provide  different  indicators  of  6ef«;een-c/ass  scatter  as  compared  to  within-class  scat¬ 
ter.  Parsons  [40:pp  176-180]  provides  an  introductory  discussion  on  separability  measures 
including  Fisher  Ratios  and  Fukunaga’s  Jl  -  J4  indicators. 

The  measure  J4  by  Fukunaga  [17]  is  defined  by 


J4  = 


trSl 

lrS2 


where,  trS\  and  trS2,  are  the  trace  of  inter-class  scatter  matrix,  and  the  intra-class  scatter 
matrix,  respectively.  The  trace  of  a  matrix  is  defined  [46]  as 


k  k 

trA  =  ^  ^  Aj 

i=l  i=l 

where  A,s  are  the  eigenvalues  of  the  matrix  A,  which  is  kxk.  The  eigenvalues  represent  the 
major  and  minor  axes  of  an  fc-dimensional  hyper-ellipsoid,  (where  A,  >  0)  in  this  case  the 
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cluster  of  feature  vectors  representing  a  class.  The  scatter  matrices,  51  and  52  are  defined 
by 

51  =  f;  (15) 

1  =  1 

52  =  f;  Piui)E  {(X  -  Mi)iX  -  MifluJi)  (16) 

i=l 

Where  M  is  the  number  of  classes  in  the  data  population.  Equation  16  is  often  represented 
as  ^ 

51  =  f;P(a;,)C,  (17) 

»=i 

=  (18) 

•'  <=1 

P{Ui):  is  the  a  priori  probability  of  class  i 
M,:  is  the  mean  of  class  i 

where:  Mo’  is  the  global  mean  of  the  whole  data  set 

Ci’.  is  the  covariance  matrix  of  a  class  i. 

N  :  is  the  number  of  feature  vectors  in  the  class 
The  larger  the  J4  separability  measure  the  more  separable  a  data  set  is.  The  break 
even  point  is  when  J4  =  1,  which  means  that  the  inter-class  scatter  is  identical  to  the 
intra-class  scatter.  This  would  stiU  present  a  non-trivial  pattern  classification  task. 

The  separability  measures  results  for  the  two  data  sets  are  shown  in  Table  2.  At  first 
inspection  both  the  Test6  and  Iris  data  appear  to  be  easily  separated  into  three  classes. 

Separability  measures  must  be  considered  carefully,  as  indicators  can  be  misleading. 
The  Iris  data  is  plotted  in  Figure  10  using  three  of  the  four  dimensions.  Note  that  two  of 
the  classes  overlap.  This  is  also  the  case  for  the  Test6  data  in  Figure  9.  Computing  the 
J4  measure  for  the  Iris  data  set  when  considering  two  classes  at  a  time  reveals  the  true 
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ill*  data 


Figure  10.  Iris  data  dimensions  2,3,4 


Table  3.  Separability  measure  J4  for  Iris  and  Test6 


Iris  J4 

(jJ2  u>3 

1  5.601  9.608 

Wi 

1  0.885 

(J2 

1 

UJ3 

Test6  J4 

Ui 

U2 

a;3 

1 

1.842 

0.598 

Wl 

• 

1 

50.105 

U2 

• 

• 

1 

U)3 

structure  of  the  data  set.  This  is  shown  in  table  3,  the  first  row  shows  that  class  one  (Iris 
Sestosa)  is  clearly  separable  from  the  other  two  classes  (Iris  Versicolor  and  Iris  Virginica). 
The  second  row  reveals  the  overlap  we  saw  (Figure  10)  between  the  other  two  classes.  The 
overall  J4  measure  provides  a  general  indicator  of  the  data  set,  however  it  is  important  to 
consider  the  classes  in  pairs  to  obtain  an  accurate  indication. 


S.9  Conclusion 

Three  speech  databases  are  used  in  this  speaker  identification  research.  The  AFIT93, 
TIMIT  and  King  data  were  all  pre-processed  identically  using  the  ESPS  speech  processing 
library.  The  cepstral  feature  vectors  from  segments  of  speech  which  had  greater  than  ten 
percent  probability  of  voicing  were  retained  as  the  feature  set.  The  data  was  partitioned 
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into  a  training  and  a  test  set  for  each  day.  The  identification  experiments  for  each  method, 
vector  quantization,  fuzzy  clustering  and  neural  network,  were  trained  using  the  data  from 
day  one  up  to  the  final  day.  The  testing  was  conducted  using  the  test  data  from  all  days. 
For  example  the  day  three  experiment  was  trained  using  data  from  day  one,  uo  and 
three.  It  was  then  tested  using  data  from  all  days,  five  for  TIMIT  and  King  and  seven 
for  AFIT93.  The  performance  is  based  on  identification  rate  since  the  three  clustering 
methods  do  not  have  a  common  distance  or  distortion  metric.  Chapter  four  presents  the 
results  of  the  experiments  conducted,  including  an  analysis  of  the  separability,  using  a 
separability  measure  developed  by  Fukunaga  [17]. 
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IV.  Results 


4.1  Introduction 

This  chapter  presents  the  results  of  the  speaker-identification  experiments.  An  initial 
discussion  on  the  separability  measures  of  the  databases  is  followed  by  the  results  for  each 
of  the  clustering  techniques. 

4.2  Separability  Results  for  Speaker  Data 

Separability  measures  were  discussed  in  Chapter  III  and  are  used  to  indicate  how 
difficult  the  classes  are  to  separate.  Table  4  lists  the  J4  separability  measure  for  each  of 
the  speech  databases.  The  table  shows  the  separability  measure  for  each  speaker  when 
compared  to  each  of  the  others  speakers  in  turn.  Since  the  matrix  is  symmetric  only  the 
upper  half  is  shown.  Recall  that  the  J4  measure  is  the  ratio  of  the  inter-class  scatter 
to  the  intra-class  scatter,  and  for  linear  separability  J4  must  be  greater  than  one.  All 
three  databases  present  a  considerable  challenge,  King  in  particular  has  a  large  number  of 
classes  that  appear  to  be  inseparable,  at  least  to  three  decimal  places.  Keeping  in  mind 
that  one  is  the  break  even  point  for  the  J4  measure,  displaying  three  decimal  places  is 
only  done  to  ensure  that  the  tables  were  not  filled  with  zero!  It  is  tempting  to  conclude 
that  the  task  of  identifying  the  speakers  in  these  databases  is  impossible  and  to  stop 
there.  Actually  this  data  provides  a  justification  for  using  clustering  algorithms  to  design 
a  speaker  identification  system.  Clustering  algorithms  attempt  to  detect  structure  in  data 
sets  which  is  difficult  to  discern.  This  is  basically  sub-partitioning  the  feature  space  into 
smaller  regions  which  (hopefully)  provide  the  separability  between  classes  that  we  require. 

The  J4  values  still  provide  useful  information  despite  the  low  values.  The  relative 
values  indicate  which  classes  are  likely  to  be  the  most  difficult  to  isolate.  The  confusion 
matrices  in  Section  4.6  confirm  the  J4  values  for  the  AFIT93  speakers.  The  first  speaker, 
cm,  is  incorrectly  identified  as  being  the  speaker  for  the  majority  of  the  other  AFIT  speak¬ 
ers.  In  the  confusion  matrix  this  is  indicated  by  the  values  in  the  first  column.  The  J4 
measures  in  Table  4  confirm  this,  since  the  relative  values  for  cm  are  some  of  the  lowest. 
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Table  4.  Separability  Measure  J4  for  AFIT,  King  and  TIMIT 


cm 

dp 

ei 

gs 

jc 

jk 

jm 

jt 

km 

me 

rm 

wg 

1 

0.009 

0.006 

0.004 

0.007 

0.007 

0.008 

0.014 

0.010 

0.008 

0.005 

0.018 

cm 

1 

0.012 

0.011 

0.013 

0.011 

0.004 

0.032 

0.006 

0.005 

0.016 

0.021 

dp 

1 

0.002 

0.009 

0.005 

0.010 

0.015 

0.010 

0.016 

0.003 

0.010 

ei 

1 

0.005 

0.008 

0.009 

0.011 

0.008 

0.011 

0.004 

0.017 

gs 

1 

0.015 

0.013 

0.009 

0.009 

0.017 

0.010 

0.028 

jc 

1 

0.006 

0.025 

0.011 

0.014 

0.005 

0.005 

jk 

1 

0.031 

0.007 

0.005 

0.011 

0.017 

jm 

• 

1 

0.024 

0.035 

0.017 

0.035 

jt 

• 

• 

1 

0.012 

0.013 

0.018 

km 

• 

• 

1 

0.019 

0.030 

me 

• 

• 

• 

1 

0.010 

rm 

• 

• 

• 

1 

wg 

spl 

sp2 

sp3 

sp4 

sp5 

sp6 

sp7 

sp8 

sp9 

splO 

spll 

spl2 

1 

0.016 

0.039 

0.006 

0.116 

0.045 

0.006 

0.023 

0.001 

0.018 

0.000 

0.000 

spl 

1 

0.009 

0.021 

0.045 

0.011 

0.007 

0.007 

0.015 

0.003 

0.000 

0.000 

sp2 

1 

0.050 

0.022 

0.002 

0.021 

0.003 

0.038 

0.005 

0.000 

0.000 

sp3 

1 

0.124 

0.060 

0.008 

0.037 

0.003 

0.030 

0.000 

0.000 

sp4 

1 

0.020 

0.079 

0.043 

0.111 

0.042 

0.000 

0.000 

sp5 

1 

0.027 

0.005 

0.045 

0.006 

0.000 

0.000 

sp6 

1 

0.012 

0.005 

0.010 

0.000 

0.000 

sp7 

• 

1 

0.024 

0.002 

0.000 

0.000 

sp8 

• 

• 

1 

0.018 

0.000 

0.000 

sp9 

• 

• 

• 

1 

0.000 

0.000 

splO 

• 

• 

• 

• 

1 

0.000 

spll 

• 

1 

spl2 

femmO 

ferhO 

fedwO 

mcmjO 

mefgO 

mhpgO 

mjlsO 

mmwhO 

mprkO 

mrtkO 

1 

0.077 

0.164 

0.016 

0.007 

0.008 

0.024 

0.147 

0.063 

0.046 

1 

0.016 

0.043 

0.111 

0.104 

0.131 

0.022 

0.008 

0.181 

1 

0.107 

0.223 

0.213 

0.257 

0.010 

0.031 

0.335 

liii 

1 

0.023 

0.020 

0.038 

0.106 

0.031 

0.064 

mcmjO 

1 

0.001 

0.013 

0.208 

0.092 

0.021 

mefgO 

■ 

1 

0.007 

0.202 

0.088 

0.020 

mhpgO 

• 

• 

1 

0.251 

0.119 

0.022 

mjlsO 

• 

• 

• 

1 

0.028 

0.323 

mmwhO 

• 

• 

• 

• 

1 

0.151 

mprkO 

• 

• 

• 

• 

• 

• 

1 

mrtkO 

4-S  Vector  Quantization  Benchmark 

The  vector  quantization  system  provides  the  benchmark  performance  for  compari¬ 
son  with  the  other  techniques.  Figure  11  displays  the  identification  rates  for  the  three 
databases.  For  all  databases  the  identification  rates  progressively  climb  with  the  number 
of  sessions  used  in  the  codebooks.  The  lower  identification  rates  for  the  King  database 
are  due  several  factors  including  dramatic  changes  in  the  recording  after  session  five,  and 
the  speech  is  conversational,  unlike  the  phonetically  balanced  sentences  used  in  TIMIT 
and  AFIT93.  The  codebooks  have  64  codewords  and  use  the  Euclidean  distortion  mea¬ 
sure.  The  performance  of  the  classifier  is  lower  when  the  test  data  is  used,  this  drop  in 
performance  provides  an  indication  of  how  well  the  classifier  can  generalize. 

4.4  Fuzzy  Clustering  Experiment 

The  UFP-ONC  based  speaker-identification  system  reported  in  these  results  used 
eight  centroids.  Figure  12  shows  the  overall  identification  rate  for  the  test  data  sets.  The 
UFP-ONC  rates  for  AFIT93  and  TIMIT  are  72%  and  67%  respectively.  The  result  for 
King  is  very  low,  this  is  most  likely  due  to  the  UFP-ONC  algorithm  not  converging  for 
speakers  sp4  and  spl2  on  session  five.  This  implementation  of  the  UFP-ONC  required 
convergence  in  500  iterations,  if  convergence  is  not  achieved  then  the  centroids  are  saved 
and  used  as  the  best  set  available.  This  is  not  an  an  adequate  long  term  solution,  however 
it  is  considered  to  be  a  reasonable  approach. 

Figure  13  shows  the  performance  of  UFP-ONC,  LEG  and  the  MLP  for  the  AFIT93 
corpus.  The  graphs  display  the  mean  identification  rate  with  the  line.  The  error  bars 
are  plus  and  minus  one  standard  deviation  from  the  mean.  Note  that  LEG  achieves  a 
mean  rate  of  94.2%  while  the  UFP-ONC  achieves  92%,  this  indicates  that  the  UFP-ONC 
can  achieve  performance  as  high  as  vector  quantization.  The  vector  quantization  system 
is  more  consistent  as  seen  by  the  smaller  standard  deviation.  Like  the  LEG  algorithm, 
the  UFP-ONC  exhibits  better  performance  as  the  number  of  sessions  of  training  data  is 
increased.  This  appears  to  confirm  that  the  long  term  statistics  of  the  speech  feature 
vectors  are  important  to  identification  accuracy. 
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4-5  ANN  Experiments 

Figure  14  displays  the  identification  rate  for  the  multi-layer  perceptron.  The  iden¬ 
tification  rates  are  significantly  lower  than  those  achieved  by  vector  quantization.  Each 
network  was  presented  with  feature  vectors  from  all  speakers  and  trained  with  one  out¬ 
put  for  each  speaker.  Had  the  cepstral  features  been  grouped  by  families  of  vowels  or 
phonemes  such  as  the  work  reported  by  Rabiner  and  Juang  [42]  the  identification  rates 
would  be  higher. 

All  networks  were  trained  with  20  nodes  in  the  hidden  layer,  for  3000  epochs.  None 
of  the  ANNs  achieved  the  desired  mean  squared  error  of  0.01,  generally  the  MSE  was  0.81 
after  3000  epochs.  The  TIMIT  networks  were  trained  for  30,000  epochs  but  still  did  not 
reduce  the  mean  squared  error  below  0.7.  To  provide  an  even  comparison  between  the 
clustering  algorithms  the  data  was  presented  in  exactly  the  same  way  each  time.  This 
presentation  of  cepstral  data  is  obviously  not  suited  for  classification  by  MLP.  Another 
factor  that  must  be  considered  is  the  training  time  required  for  neural  networks.  The 
King  and  AFIT93  databases  required  four  days  of  processing  for  3000  epochs  on  both  the 
IBM  RS6000  and  the  Silicon  Graphics  Iris  4D.  This  made  it  extremely  difficult  to  conduct 
multiple  tests  which  may  have  provided  better  results. 
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Figure  13.  AFTT  Identification  Rate 
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4.6  Confusion  Matrices 

Confusion  matrices  for  the  AFIT93  corpus  were  generated  from  the  results  of  the 
day  one  and  the  day  seven  codebook  tests.  The  confusion  matrices  are  shown  in  Tables  5 
and  6.  The  vertical  axis  of  the  matrix  represents  the  true  identity  of  the  speaker,  while 
the  horizontal  axis  represents  the  decision  made  by  the  classifier.  For  example,  looking  at 
Table  5,  the  second  row  of  the  LBG  matrix  shows  that  speaker  2  was  identified  six  times 
as  speaker  1,  twenty-three  times  as  speaker  2  (correct),  and  once  each  as  speaker  3  and 
speaker  5.  In  addition  to  showing  that  the  correct  identification  rate  for  speaker  2  was 
74.2%  (||)  the  confusion  matrix  indicates  which  person  was  selected  when  an  incorrect 
decision  was  made. 

The  multilayer  perceptron  appears  to  have  split  the  speakers  into  two  classes.  It 
appears  that  attempting  to  train  a  multilayer  perceptron  using  all  feature  vectors  from  all 
speakers  is  not  a  suitable  method.  By  choosing  a  subset  of  speech  features,  such  as  the 
formants,  ANNs  has  been  shown  to  perform  extremely  well  [42] 

The  UFP-ONC  achieves  comparable  performance  to  vector  quantization.  Note  that 
both  methods  show  strong  diagonals  in  the  matrix,  the  results  for  day  seven  are  especially 
pronounced.  The  confusion  matrices  provide  an  easy  method  of  determining  whether  the 
LBG  and  UFP-ONC  could  be  used  to  complement  each  other,  however  the  confusion 
matrices  also  indicate  that  the  vector  quantization  and  the  fuzzy  technique  make  similar 
errors. 

4.7  Conclusion 

This  chapter  introduced  separability  measures  and  provided  an  analysis  of  the  sepa¬ 
rability  measure  for  each  of  the  speech  databases  using  Fukunaga’s  J4  measure.  The  sep¬ 
arability  measures  indicate  that  all  three  databases  pose  a  signiiicant  problem  for  speaker 
identification  using  any  of  the  statistical  based  classifiers.  Clustering  techniques  are  con¬ 
sidered  to  the  most  suitable  choice  to  form  the  basis  of  a  speaker  identification  system. 
The  identification  rates  for  each  of  the  three  techniques  was  reported,  with  both  the  LBG 
vector  quantization  and  the  fuzzy  UFP-ONC  achieving  rates  of  more  than  90%  on  indi- 
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Table  5.  AFIT  Day  One  -  Confusion  Matrices  for  LBG,  UFP-ONC,  MLP 
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27 
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• 

1 

6 

• 

1 

1 

3 

• 

24 
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1 

1 

2 

1 

. 

24 

1 

, 

1 
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. 

28 

. 

15 

1 

3 

• 

1 

1 

2 

7 

. 

3 

. 

1 

• 

• 

17 

UFP-ONC 

29 

• 

•  . 

• 

1 

, 

. 

. 

4 

24 

•  • 

» 

3 

1 

2 

10 

1 

• 

4 

2 

1 

28 
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• 

1 

9 

5 
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• 

5 

5 
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3 
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7 
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• 

28 

5 
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3 
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• 
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11 
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• 
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• 
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7 
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3 
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6 

7 
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Tablt  6.  AFIT  Day  Seven  -  Confusion  Matrices  for  LBG,  UFP-ONC,  MLP 
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viduals.  LBG  is  more  consistent  than  the  UFP-ONC  at  this  time,  however  the  UFP-ONC 
has  achieved  very  good  performance  with  only  eight  cluster  centers  in  the  speaker  code¬ 
books  as  compared  with  64  codewords  in  the  vector  quantizer  codebooks.  However  this 
implementation  of  the  UFP-ONC  is  not  yet  robust,  a  number  of  problems  affect  its  per¬ 
formance.  This  preliminary  investigation  in  speaker  identification  using  fuzzy  techniques 
indicates  that  further  research  into  fuzzy  based  methods  is  warranted. 
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V.  Conclusions 


In  this  research  the  Unsupervised  Fuzzy  Partition-Optimal  Number  of  Classes  al¬ 
gorithm  was  implemented  and  applied  to  speaker-identification.  The  performance  of  the 
UFP-ONC  is  compared  to  LEG  vector  quantization  and  the  multilayer  perceptron. 

This  research  demonstrated  that  fuzzy  clustering  methods  can  achieve  the  same 
or  higher  rates  of  identification  on  individuals  in  a  database.  The  UFP-ONC  clustering 
algorithm  provides  a  very  effective  means  of  developing  fuzzy  speaker  codebooks.  The 
fuzzy  codebooks  achieve  very  good  identification  rates  with  fewer  centroids  than  used  in 
vector  quantization.  This  comes  at  the  expense  of  increased  computation  during  training 
and  additional  storage  for  the  fuzzy  covariance  matrices. 

The  present  implementation  of  the  UFP-ONC  requires  more  research  to  achieve  the 
robust  operation  of  the  LEG  based  vector  quantizer.  Issues  that  need  to  be  addressed 
include  : 

•  How  to  prevent  repeated  centroids  in  the  Fuzzy  Ir-means  algorithm.  The  current 
implementation  of  the  UFP-ONC  does  not  test  for  coincident  centroids.  Coincident 
centroids  prevent  correct  operation  of  the  classifier  by  assigning  equal  memberships 
to  each  of  the  clusters. 

•  How  to  adapt  the  validity  criteria  to  provide  meaningful  indicators  for  speech  data. 
The  cluster  validity  measures  were  not  reliably  generated  when  using  speech  data. 
When  applied  to  simpler  data  sets  such  as  the  Iris  data,  Test6  or  even  RGE  vectors 
from  images,  the  measures  were  generated  consistently. 

•  An  evaluation  of  centroid  initialization  techniques.  The  accuracy  of  the  FMLE  al¬ 
gorithm  depends  on  the  accuracy  of  the  centroids  found  by  the  FKM.  The  FKM 
also  benefits  from  “good”  initialization.  This  research  used  two  techniques,  the  first 
calculated  the  initial  centroids  by  adding  random  vectors  to  the  mean  of  the  data 
set.  The  second  generated  the  initial  centroids  by  randomly  selecting  ten  samples 
from  the  data  set  and  calculating  the  mean.  Both  performed  these  performed  well, 
however  there  may  be  a  more  appropriate  method  which  can  be  incorporated. 
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•  The  development  of  alternate  fuzzy  classifiers  may  provide  more  improve  the  identi¬ 
fication  rates  to  the  levels  attained  by  vector  quantization. 

The  objective  of  this  research  was  to  provide  and  evaluation  of  clustering  techniques 
to  speaker  identification.  This  objective  has  been  met,  and  the  results  indicate  that  further 
application  of  fuzzy  based  algorithms  is  warranted. 
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Appendix  A.  TIM  IT  Sentences 


The  table  below  shows  some  of  the  phonetically  balanced  sentences  used  in  the  TIMIT 
data  base. 


sal 

She  had  your  dark  suit  in  greasy  wash  water  all  year 

sa2 

Don’t  ask  me  to  carry  an  oily  rag  like  that 

sx22 

When  all  else  fails,  use  force 

sx40 

Stimulating  discussions  keep  students’  attention 

si806 

You  need  answers  to  four  important  questions 

si912 

You  may  amaze  yourself  and  acquire  a  real  knack  for  it 
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Appendix  B.  Program  Listings 


The  following  programs  axe  written  using  ANSI  C  structure.  The  programs  use  no 
specialized  interface  or  graphics  and  have  executed  successfully  on  the  following  platforms, 
SUN  Sparc  2,  SUN  Sparc  10,  IBM  RS6000,  Silicon  Graphics  Iris  4D  and  IBM  PC  (386). 

The  code  makes  extensive  use  of  structures  and  functions  for  ease  of  modification 
and  versatility.  The  library  FuzzLIB.c  provides  the  fuzzy  A-means,  the  fuzzy  maximum 
likelihood  estimation  and  the  UFP-ONC.  All  arrays  and  matrices  are  coded  using  the 
Numerical  Recipes  in  C  [41]  format. 

Also  included  is  a  MATLAB  [33]  script  file  which  provides  a  simple  implementation 
of  the  FKM  and  UFP-ONC  code.  This  code  was  used  in  verification  and  is  extremely  slow 
for  all  but  small  data  sets. 
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B.l  Separability  C  Code 


h 


ProgTun  N4m«  :  fuku.c 

D««cription  :  C4tcu)4te«  Sep4r4biUty  MeMure  J4,  ««€  PUKUNAQA  1973 
Author  :  D.  Ne4l«  Prescott,  (dpreacoiO4fit.4f.mil) 

University  :  USAF  Institute  of  Technology 

Dote  ;  P£B  9< 

Other  Code  :  4.  Numericnl  Recipes  in  C  (3nd  Ed) 

Note  :  If  you  don't  hove  NRC  then  just  reploce  the  OSQR  commends  with  * 
MACRO  for  squaring  two  double  precision  numbers. 

Input  Piles  :  The  data  file  in  the  following  format.  The  leading  comments 
are  not  included. 


Line  1:  Number  of  dimensions  per  feature  vector 
Line  2:  Number  of  classes 
Line  3:  Number  of  feature  vectors  in  class  1 
Line  4:  Number  of  feature  vectors  in  class  3 

Line  5:  ...  ...  . 

Line  x:  ...  ...  . 

Line  z:  ...  ...  . 

Line  N:  Number  of  feature  vectors  in  last  class 
Line  N-f  I:  data  for  class  1  one  feature  vector  per  row 

Line  z;  ...  ...  . 

Line  z:  ...  ...  . 

Line  :  data  for  class  3  one  feature  vector  per  row 


NOTE  :  The  sizes  of  the  covariance  arrays  is  set  up  for 

maximum  sizes  of  20  dimensional  feature  vectors  and 
a  maximum  of  12  classes.  The  ’’zero  th"  element  of 
all  arrays  is  not  used. 


#include  <stdio.h> 

#include  <stdlib.h> 

#include  <string.h> 

#include  <math.h> 

#include  <errno.h> 

#include  ’’nrutil.h” 

void  checkJnput..argumeBts(int  emdJine,  int  req^rgs); 

typedef  struct  statistic^arams{ 
double  mean[21]; 
double  var{2l]; 
int  samples; 

}  STATJARAMS; 

typedef  struct  covar^>arams{ 
double  covariance[21][21]; 

}  COV-PARAMS; 

/e  ssssBaswsssBswasasssa  variables  bssbsssssss3sbbsbss= 

int  dim,  numjipkrs,  i,  j,  k,  p,  tot; 

STATJ>ARAMS  spjtat[30]; 

STATJ>ARAMS  global-stat; 

COVJ’ARAMS  spjcov[30]; 
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COV  JARAMS  Sl^ov.  S3^ov; 

doabU  X«abV[3l],  J4,  J4mtx{14][l4],  trSl,  trS3,  tmpj 
cb4r  fn*m«{40]; 

PILE  •dftlAJii; 

in4iii(ial  argc,  char  •argvQ) 

{ 

/«  Read  in  the  header  anmber*  :  dimeaiion,  aumber  of  epeakete. 
aad  feature*  per  •peaker- 

. V 

cbeckiaputj*rguineoU(arge,  3); 

•priotf(faame,  ^gv(l]); 

dataJaafopeB(fBame/*r'*); 

if(!dalauB) 

{ 

priBtf(**Caat  epea  the  file  faame); 

exit(l); 

} 

ftcaBf(dataiB,  "RdRd'*,  dedim,  drnum^pkr*); 
for(i  s  1;  i  <  aum^pkre;  1++) 

{ 

fscaBf(dataJB,  "Rd”,  de«p^tat[i].ftample*  ); 
for(j  SB  1;  ]  <  dim; 

{ 

•p^tat(i].meaa[j]  s  0.0; 
sp««tat(ij.var[j]  a  0.0; 
globaljitat.meaB[j]  a  0.0; 
global^tat.var(j]  a  0.0; 

} 

} 

/*  Read  ia  the  data  and  calculate  the  laeaa*  and  vartaace* 
for  each  of  the  tepaker* 

.  V 

tot  s  0.0; 

for(i  a  1;  i  <  numj»pkr«;  i+*f ) 

{ 

for(i  s  1;  j  <  •pjitat[i]. sample*;  j++) 

{ 

for(k  a  1;  k  <  dim;  k+  +  ) 

{ 

fscaof(dataaa,  '’%ir,  detmp); 

•p^tat(i].mean[k]  'fa  tmp; 

•pjtat[i].var[k]  ^fa  DSQR(tmp); 
global^tat.meaaOt]  +a  tmp; 
global.jtat.var[k]  +a  DSQR(tmp); 


> 

tot++; 

} 


/«  Complete  the  final  calculation  of  mean  and  var  for  each  claet 

.  •/ 

for(i  ss  1;  i  <  ttumjpkre;  i+  +  ) 

{ 

foi(k  »  1;  k  <  dim;  k++) 

{ 

•putat(i].mean[k]  s  •p_«tat[t].mean[k]/ipjtat[i].«amplee; 

•putat|ij.var[k]  s  (•p^tat[i].var[k]/ipjtat[i]. sampler)  —  DSQR(spjtat[i].mean[k]); 

} 

} 

for(j  =  1;  j  <  dim;  i++) 

{ 

global jtat.mean[i]  s  global^tat.mean{j]/tot; 

global jtat.var[j]  s  (global-stat.var[j]/tot)  —  D5QR(global jtat.mean[j]); 

} 
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/#  c«lcul4te  th«  cov4n4iice  m4trts  of  (he  cl4««  m«4B«,  Si¬ 
lt  ie  4«ettiii«d  •qu4l  4  priori  for  •pe4ker«. 

.  V 

for(i  a  1;  i  ^  dim;  i^4) 
for(j  •  1;  j  ^  dim;  j++) 

Sl^ov.cov4ri4nce[i][i]  m  0.0; 

for(i  a  1;  i  ^  numapkre;  i+*f ) 

{ 

fot(j  a  1;  j  ^  dim;  j  +  +  ) 

X«ttbV[j]  a  •p^t4t(i].me4n[i]  —  (lob4l^t4(.me4By]; 

for(j  ■  1;  i  <  dim;  i  +  +  ) 
for(k  a  1;  k  <  dim;  k+4-) 

Sl^ov.cov4ri4iice[j](k]  XettbV[j]«X*ubV(k]; 

} 


/•  Re4d  in  the  d4t4  4(4iD  end  C4icul4(e  the  INTRA-CLASS  covBri4iice« 
4Bd  S3. 


V 


rewiiid(d4t4iB); 

fec4Bf(d4t4a&,  ’’Kd^d**,  ici,  dej); 
for(t  a  1;  i  <  Bum.^pkrA;  i+*f ) 
f«C4af(d4t4Ja,  "9(d",  tel  ); 


/•  Read  ia  the  date  aad  calculate  the 


for(i  a  1;  i  <  dim;  i4’  +  ) 
for(j  »  1;  i  ^  dim;  i++) 
S3.dCOV.cov4ri4Bce[i][j]  a  0.0; 


covariaace  matrix 

V 


(ar(i  s  1;  i  <  num-jpkrs; 

{ 

for(j  »  1;  j  ^  dim;  i++) 
for(k  »  1;  k  <  dim;  k++) 

sp.xov[i].cov4ri4ace(j][k]  a  0.0; 

for(j  »  1;  j  ^  •p.j(4t(i].e4mplee;  j+'f) 

{ 

for(k  s  1;  k  <  dim;  k++) 

{ 

fiC4Bf(d4t4Ja,  ”Klf’,  &XsubV(k]): 

XtabV(k]  a  XeubV(k]  —  •p.jt4t[ij.roe4B[k]; 

} 

for(p  s  1;  p  <  dim;  p++) 
for(k  a  1;  k  <  dim;  k++) 

•p^ov(i].cov4ri4Bce[p][k]  4-^  XeubV[p]*XeubV[k]; 

} 

forO  «  1;  j  <  dim;  j+4-) 
for(k  a  1;  k  <  dim;  k*f  +  ) 

{ 

•pjcov[i].cov4ri4acey](k)  a  fpjcovyi.covariaaceyjlkj/ipjtattil.eamplee; 
S2jcov.cov4ri4Bcey][k]  4’*^  •pjcov[i].cov4ri4ace[j](k|/num^pkre; 

} 

} 

fclo«e(d4t4Ja); 


/«  Calculate  the  trace  of  SI  and  of  S3,  then  calculate  J4. 
'^or  all  claesee. 

. </ 

trSl  •>  0.0; 
trS3  m  0.0; 

J4  a  0.0; 

for(i  a  1;  i  <  dim;  i-l-+) 

{ 

trSl  +8  Sl.cov.covariance[i][i]; 
trS3  +8  S3aCOV.covariance[ij[ij; 

> 

J4  a  trSl/frS3; 

priatfC’Separability  Meaeure  for  [%•]  i*  9(lf"n”,fname  .  J4); 
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printf(‘*trSl  •  1l4.3«**UrS3  m  K4.3«“ik",  (rSl.  trSd); 


/•  CU^uUU  th«  trftc«  of  SI  and  of  S3,  khon  cateuUu  J4. 

For  aU  c1m«o«.  Nolo  khak  only  Ibo  calculakion*  which 
affoct  the  diagonal  havo  boon  computed  khit  timo. 

.  V 

for(i  *  1;  i  <  num.apkro;  i-f  4-) 
for(j  m  li  i  ^  num^pkro;  i44) 

J4mtx[i](j]  m  0.0; 

for(i  w  1;  i  <  numapkro;  i44) 

{ 

for(i  ■  (i41):  j  <  num^pkro;  j44) 

{ 

trSl  m  0.0; 
trS3  a  0.0; 

for(k  a  1;  k  <  dim;  k4  +  ) 

{ 

X«ubV[k]  a  (•p^tat[i].moan[k]  4  «p.i«tak(j].moan[k|)/20; 

irSl  4*  OSQR(<pjtal[i].meatt{k]  -  X«ubV(k])  4  DSQR(*pjial(i].meaa(kJ  -  XoubVikJ); 
trS3  4*  (•p.xov[i].eovananco[k](k]  4  «pjcov[j].covananco(k](k)); 

) 

a  trSl/trS3; 

} 

} 

/•  Print  it  out  ready  for  a  Latex  table 
Juot  copy  the  entire  output  into  a  Latex 
document  and  the  table  U  ready  to  go 

. V 

printf(”  “  “begin-t  able  “  “n” ); 
printf(**  '*'‘begin-center"“n” ); 

printf(”  '^^begin-tabular" - "); 

for(i  a  1;  i  <  num^pkro;  i44)  printf("c”); 

printf('‘ - "  “**hltne  ““hline  “n"); 

for(i  a  1;  i  <  num^pkro;  i44) 
if(i  numjipkrt) 

printf(”l“‘*omega  -Wd"l  Sc  ",  i); 
eUe 

printf(''$"**omega-9(d*'l  ",  i); 
printf("  ********  ““hUne  *'n”); 

for(i  a  1;  i  <  numapkro;  i44) 

{ 

for(j  a  1;  j  <  num^pkre;  j44) 

{ 

if(J4mtx[i][j]  aa  0.0) 
printf("l*‘"cdotl"t”); 

elee 

printf(’'%4.3f'*t”,J4mtx(i][j]); 

if(j  aa  num<^pkr«) 
printf(’’“"*'"  ‘’n”); 

else 

pritttf("&  "); 

} 

} 

prinlf(”““hline  **‘‘hline  “n"); 
printf("  ""end-tabular  "  "n”  ); 
printf("  “"end-center  “"n"); 

printf('*""label-Pukungaga  J4  meaoure  for  file  9(s*'“n",  fname); 
printf(’*  ""end-table"  “n"); 


} 

void  checkJnput^rgumento(int  cmdjine,  int  req^rgo) 

{ 

/•  This  function  ensure*  the  correct  number  of 
command  line  argument*  are  present. 

Neale  Prescott  OCT  93 

V 


if(cmdjine  ^  req^rgs) 


prinlf(’"*ii&RROR  Comin4nd  l)n«  4rgumeiit«  iii€oinpl«t4**n'*)i 

•«»(»); 


{ 

) 

} 
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B.2  UFP-ONC  C  code 

B.2.1  Main  Program  -  FuzzCl.c. 
/• . 


Program  Nam«  :  PussCl.c 

Ootcription  :  This  program  provide*  a  number  of  Puasy  Ctueicring 
algorilhme,  the  Pussy  K*m«aa«  (PKM)  and  the  Puasy  Uasimum 
Likelihood  Estimation  (PMLE).  These  two  are  combined  to  form 
the  Unsttpervised  Pussy  Partitioa>Oplimal  Number  of  Classes 
(UPP*ONC)  proposed  by  I. Oath  and  B.Oeva  IEEE  Transaction* 
on  Pattern  Analysis  and  Machine  Intelligence,  Vol  11.  No. 7, 

July  1089. 

The  UPPoONC  will  be  used  to  determine  the  number  and  location 
of  clusters  in  the  data  set.  This  data  will  then  be  passed  to 
the  Possibilistic  C-Means  (PCM)  proposed  by  Krishnappuram 
and  Keller,  IEEE  Transactions  on  Pussy  Systems,  Vol  1.  Ho. 2, 

May  1993,  algorithm  for  final  determination  of  the  cluster 
centres. 

The  intended  purpose  of  this  rather  complicated  process 
is  to  cluster  speech  data  for  a  number  of  speaker*  involved 
in  machine  based  speaker  identification  experiment  for 
thesis  work  by  Neale  Prescott. 

The  fussy  clustering  approach  is  to  be  compared  to 
a  Vector  Quantisation  based  Speaker  ID  system. 

If  the  number  of  clusters  is  known  then  a  faster  method 
is  to  use  the  basic  Fussy  K^means  and  then  the  PCM.  In  this 
case  the  aim  is  to  remove  the  heuristic  way  in  which  the 
number  of  clusters  (or  codewords)  is  often  chosen. 


Author  :  D.  Neale  Prescott,  (dprescotOafit.af.mil) 

University  :  USAF  Institute  of  Technology 

Date  :  OCT  93,  30  JAN  94 

Other  Code  :  a.  Numerical  Recipes  in  C  (3nd  Ed) 

b.  FussLIBh 

c.  FussLIB.c 

Input  Piles  :  datable,  setup-file 

Output  Files  :  clusterJiles,  performanceJile 

References  :  See  Fusslib.c  for  the  list. 

. M 


#include  <stdio.h> 
#include  <stdlib.h> 
#inclade  <string.h> 
#include  <math.h> 

#include  ’’nrutil.b” 
#inclade  **nr.h’' 
#include  ’'FussLIB.h” 

#ifdef  DIAO 

#define  PRINT -DIAG  1 

#else 

#define  PRINT  J>IAG  0 
#endif 


/*  wwswssasBaswwBBaiaiMsa  variable*  WMWWwwwwwwwwBVBBBiWwaiwaiBaiaiat  */ 

int  status,  i; 

C.JARAMS  config^arams; 

F.PARAMS  feat-params; 

M-PARAMS  mtx^arams; 

S^ARAMS  stats-params; 
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P^ARAMS  perforin4>*r&fn«i 


PILE  •clutterJa,  •clutter^ul,  •p«rf^uli 

m»in(int  ftrgc,  ch«r  •ftrgvQ) 

{ 


re4<ljconfigur4tion^U(ieconfig^Ar*m«,  lrfe«t^4r4mi.  4rgc.  4rgv); 
m4ke^4tnce*(&config^4r4m*,  &intx.p4r&m<,  &it4is.p4r4ro«,  ^perfons^Aixm*); 
lo4dJe4ture«(&coiiflg4>4r4fni,  &mtx.p4r4ina); 

getjd4t4^t4ts(&coafig-p4r4m*,  &mtx-p4r4m<,  &«(4t«^4r4m«); 

•witch(config^4r4m».FKM  jDr.UPP) 

{ 

C4««  'U'  :  p«rforinJLJFP.ONC(&config^4r4m<,  ^mtx^4t4nu.  ^fe4t^4r4m«,  i:ftl4tft_p4rAm«.  i/^perIorm4>*r4m«); 
bre4k; 

C4«e  *F'  ;  initi4lUextu«ter«(Xeconfig4>4r4m4,  &mtx4»4r4m*,  ic$t4tft4>4r4m«); 

C4lcJu»yJcxne4ii«( jzconfig^4r4m«,  ^mtx^arxms,  iefe4t^4r4m<); 

•et^uaayxov-id«ntity(iemtx4>4r4m«,  &fe4t^4r4ro«); 
for(i  s  I'li  <  coRfig^4r4m«.fin4lxlu«ter«;  i++) 

print  JNiaxyxov4ri4iice(Jlemtx^4ram«,  ^fe4t4>4r4m«); 

•4v«x«ntroid«(&fntx.4>4r4infl,  defeat ^aramA)^ 
break; 

} 

if(PRlNT-DIAG)  print-meinber«hip«(  deintx.paramt,  defeat-params); 
deletejmatriceA(decoQfig^4ramA,  demtx4)araniA,  deetatA-parame,  deperform.4>arams); 

returii(O); 

} 
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B.2.2  Header  file  -  FuzzLIB.h. 


/• . 

PuBiLlB.h 

Cod«  :  ANSI  C  h«4der  ttle 
Author  :  D.Noolo.Preccott 

Purpo«o  :  Puiiy  Clu«t«ring  function*  for  FKM,  UFP-ONC  oad  FVQ 

Dote  ;  30  Jon  94 

Place  :  AFIT.  Ohio,  USA 

.  V 

/•  «BBsasisn>ssxBxas»ataa  etructure*  BBBaiasaiaaBSBssBBSBBBSBBBasBs 

typedef  etruct  configttration4>ar4m*{ 

char  coinment[80];  /•  A  tingle  line  of  comment  it  retained  V 

char  data^aroe[30];  /•  input  ■  data  file  name  */ 

char  ittitialxlu*ter[30];  /•  input .  initial  clutter  data  OPTIONAL  V 
char  clutterjiame[30];  /•  output  •  clutter  centre*  data  •f 
char  perform jiame[30]i  /•  record  of  performance  «/ 
char  FKMj>r-UFP;  /•  select  -  FKM  or  UFP-ONC  ^ 

int  ttumber.vector*;  /•  number  of  input  data  points  ^ 
int  featuret4>er.vectori  /«  dimension  of  each  input  vector  «/ 
int  initial^lutterti  /•  number  of  initial  cluster*  DEFAULT  3  •f 

int  final^luttert;  /•  Maxmimum  numbei  of  clutter*  to  be  used 

double  fuaainetti 

double  stopjcriteria;  /«  stopping  point  Tor  MAX.ij(oid.ij  •  new.ij)  «/ 

int  max Jterationt;  /a  If  FKM  doe*  not  reach  stopjcriteria  of 

}  C  JARAMSi 

typedef  struct  vector^aramt( 
int  num.vcts; 
int  numJtrs; 

int  num^entroidt;  /•  Number  of  centroids  in  use  at  a  given  time  V 

}  FJ^ARAMS; 

typedef  struct  matrix4>arams{ 

double  •efeat.mtx;  /«  data  samples  •/ 

double  eedist  jntxi  /«  distance  from  each  sample  to  each  centroid  «/ 

double  eemships^tx;  /•  membership  of  each  sample  to  each  class 

double  aaold jnemberjmtx;  /«  previous  iteration’s  membership 

double  eecentroidjntx;  /•  locations  of  each  centroid  *f 

double  eeP^ovar^tx; 

double  aPRIORI; 

double  F^et; 

}MJ>  ARAMS; 

typedef  struct  *tati*tics-paramt{ 

double  amean.vct;  /•  mean  vector  of  entire  data  set  «/ 

double  evar.vct;  /•  variance  vectorof  entire  data  set  of 

}S^ARAMS; 

typedef  struct  performaoce4»arams{ 

double  eFuzxJlV;  /«  Fuzzy  hypervolume  «  small  is  good  V 

double  eAvJDensity;  /e  Average  cluster  density  -  big  is  good 

double  ePartition-Density;  /•  Partial  cluster  density  >  big  is  good  s/ 

double  Si; 
double  S2; 

}P-PARAMS; 

void  checkJnput^rgumentt(int  crodJine,  int  req^rgs); 

FILE  •open^le-read(char  nameQ); 

void  intjchecluange(int  subject,  int  low,  int  high,  char  ewhere); 

void  floatjcheckj’ange(float  subject,  float  low,  float  high,  char  ewhere); 

void  zerojxiatrix(float  eezmtx,  int  rows,  int  cols); 

void  zero.vector(float  ezvct,  int  cols); 

void  zeroulmatrix(double  eezmlx,  int  rows,  int  cols); 

void  zerojdvector( double  ezvct,  int  cols); 

void  read jconfiguration^le(C J’ARAMS  econfig^,  F-PARAMS  efeat^,  int  cmdJine,  char  scrndjiarneQ); 
void  makejnatricet(CJPARAMS  econfig^),  MJ’ARAMS  ematrix^),  S-PARAMS  Mtats-p,  PJ’ARAMS  eperform^)); 
void  delete jnatrices(C^ARAMS  econfig^),  MJ’ARAMS  ematrix^),  S^ARAMS  sstats^,  PJ’ARAMS  eperform^)); 
void  load Jeatures(C J’ARAMS  econflg^,  M-PARAMS  ematrix^)); 

void  loadjcentroids(C J*ARAMS  econfig^),  MJPARAMS  ematrix^,  double  e*cov^>,  double  «det^,  double  saprior-p) 
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void  got jd4t4^l4t«(C J’ARAMS  •coafig4>,  M^ARAUS  •m4trix-p,  SJ^ARAMS  Mt4tf^); 

void  porformJUPP jONC(C  J*ARAMS  •config4>,  M-PARAMS  •m4trix^,  P  J*ARAMS  •f«4t^k,  S  J’ARAUS  *«t4t^,  P_PARAUS  •pcrform^)^ 
void  iiiiti4ti*«^lu«ter«(C^ARAMS  4config^,  MJ^ARAMS  •m4trix^,  SJ^ARAMS  ••t4t^)i 

void  C4lc Juasy Jijaa«4ii«(C^ARAMS  •config.p,  MJ‘ARAMS  •in4trix^,  PJ*ARAMS  •fext^); 

void  copyunembor JojOLD(MJ*ARAMS  •mxtrix^,  PJ*ARAldS  •f«4t^); 

void  C4lcul4teuii*t JPKM(C J^ARAMS  vconfig^,  M-PARAMS  Anixtrix^,  PJ^ARAMS  *(041^); 

void  compat«jn«inber^KM(C.PARAMS  vcoBfig^,  M J*ARAMS  •mxirix-p,  P J^ARAMS  *1041^); 

void  comput«^entroidtJ^KM(C-PARAMS  vcoBfig^,  M^ARAMS  #m4trix.p,  F-PARAMS 

double  compute^bjectiveJF*KM(MJPARAMS  emxtrix-p,  F-PARAMS  •fe4t4>)i 

double  compute JlVQjobjeclive-FKM(M-PARAMS  •mxtrix-p,  P-PARAMS  efext-p); 

void  cxlc  Juasy jnle(C J^ARAMS  ecoofig-p,  M-PARAMS  •matrix^,  P-PARAMS  •fe4t4>,  P-PARAMS  •perform4>4r4ms); 

void  ioitixliae J'MLEjnemb«(C-PARAMS  ecoofig^,  M^ARAMS  emxtrix-p,  P-PARAMS  efext^); 

void  «et Juxay^ovJdeotity(M-PARAMS  •mxtrix^k,  PJ*ARAMS  efext-p); 

void  cxlcjcentroid  Jrob(M.PARAMS  •mxtrix-p,  F^ARAMS  efext-p,  iot  clutter); 

void  C4lc.F^ov4r(M-PARAMS  emxtrix^,  F.^ ARAMS  efext^y,  iui  clutter); 

void  C4lcjditt.JMLB(M J*ARAMS  emxtrix-p,  P-PARAMS  efext^.  int  clutter,  P-PARAMS  eperform^)); 

void  priot-Puxayxov4ri40ce(M-PARAMS  ematrix-p,  PJ*ARAMS  efe4t4>); 
void  priot jnemberahipt(M-PARAMS  emxtrix^,  F-PARAMS  efext^)); 
void  •4ve^eotroidt(M.PARAMS  ematrix^),  F-PARAMS  efcat-p); 
void  t4vejditt4ncet(C-PARAMS  econfig-p,  M-PARAMS  ematrix-p); 

void  calc-clutt  J*erform(M.PARAMS  •matrix-p,  P-PARAMS  eperform-p,  P-PARAMS  efeat^)); 
void  ttore-perform4nce^at4(C-PARAMS  ecoofig-p,  P-PARAMS  eperform-p); 
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B.2.S  Fuzzy  C  Library  -FuzzLIB.c. 


/• 


Program  Namo  :  PuasLIB.c 

Deicription  :  This  program  provide*  a  numbor  of  Fussy  Cluttering 
algorithm*,  tho  Fussy  K*mean«  (PKM)  and  the  Pussy  Masimum 
Likelihood  Estimation  (FMLE).  These  two  are  combined  to  form 
the  Utt*upervi*ed  Fussy  Partition>Optimal  Number  of  Cla««e* 
(UPP’ONC)  proposed  by  I.Qath  and  B.Qeva  IEEE  Transactions 
on  Pattern  Analysis  and  Machine  Intelligence,  Vol  11.  No. 7. 

July  1989. 

The  intended  purpose  of  this  rather  complicated  process 
is  to  cluster  speech  data  for  a  number  of  speakers  involved 
in  machine  based  speaker  identification  experiment  for 
thesis  work  by  Neale  Prescott. 

The  fussy  clustering  approach  is  to  be  compared  to 
a  Vector  Quantisation  based  Speaker  ID  system. 


Author  :  D.  Neale  Prescott  (dprescotOafit.af.mil) 

University  :  USAP  Institute  of  Technology 

Date  ;  1993.1994 

Other  Code  :  a.  Numerical  Recipes  in  C  (2nd  Ed) 
b.  FussLIB.h 

References  :  1.  Unsupervised  optimal  Fussy  Clustering  by 

I. Gath  and  B.Geva,  IEEE  Transactions  on  Pattern 
Analysis  and  Machine  Intelligence,  Vol  11,  No. 7, 

July  1989. 

2.  Pattern  Recognition  with  Fussy  Objective  Function 
Algorithms  by  James  Besdek,  Plenum,  1987  (2nd  print) 

3.  Fussy  Clustering  with  a  Fussy  Covariance  Matrix 
by  D.E. Gustafson  and  W.C.Kessel 

IEEE  CDC,  San  Diego,  pp  761.76(i,  Jan  1979 

4.  Fussy  Models  For  Pattern  Recognition, 

Methods  that  Search  for  Structures  in  data. 

Editors  J.C. Besdek  and  S.K.Pal 

IEEE  Pres*  1992  (Selected  reprints) 


«/ 


^include  <stdio.h> 
#include  <stdlib.h> 
#include  <striiig.h> 
#include  <matb.h> 
#include  <errno.h> 

#include  "nrutil.h’' 
#include  "nr.h" 
#include  ’'FussLIB.h" 


#tfdef  DIAG 

#define  PRINT JDIAG  1 

#el*e 

#define  PRINT J)IAG  0 
#endif 

/•  sssssesssssssssssssssssss  functions  ss«sss=ss: 

void  checkJnput^rguments(int  cmdJine,  int  req^rgs) 

{ 

/e  This  function  ensures  the  correct  number  of 
command  line  arguments  are  present. 

Neale  Prescott  OCT  93 
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{ 

printf(““BERROR  Command  line  ergumenU  incompleie^n**); 

) 

) 


PILE  •open^lej<»d(ch»r  nuncQ) 

{ 

/*  Thi*  function  opont  nn  ASCII  file  for  rending. 

Nenle  Prescott  OCT  93 

V 

FILE  efp: 

fpsfopen(nnme,  *'  r"  ); 

if(!fp) 

{ 

printf('*ERROR  Cent  open  the  file  9(s*‘n”,  nntne); 
exit(l); 

} 

returii(fp); 

}  ^ 

void  intefbock J4age(int  eubjoct,  int  tow,  int  high,  cher  ewhere) 

{ 

/•  Thi«  function  check*  the  r^nge  of  nn  integer.  The  USER 

ipeciflee  the  LOW  «nd  HIGH  vnlue*.  The  u«e  of  WHERE  provide* 

«  wny  to  write  out  n  epeciflc  meeenge  to  aid  debugging. 

Ne&le  Prescott  OCT  93 
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if(  eubject  <  low  )  printf('’Nuinber  too  LOW  :  [%d]  Kd  :  low,  subject,  where)^ 

if(  subject  >  high  )  printf(*’Number  too  HIGH  :  [Kd]  9(d  :  9(s“&'\high,  subject,  where); 

} 

/•  saaassaBasasaasaeasessssssaesssrssssssBSSSzsaesessssssssssssesassssssssssss 

void  dottblexheckjr*nge(double  subject,  double  low,  double  high,  cbnr  ewhere) 

{ 

/e  This  function  checks  the  renge  of  4  double  number.  The  USER 
specifies  the  LOW  end  HIGH  volues.  The  use  of  WHERE  provides 
4  W4y  to  write  out  4  specific  messsge  to  4id  debugging. 

Nenle  Prescott  OCT  93 
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if(  subject  <  low  )  printf(" Number  too  LOW  :  [9(lf]  96lf :  %s**b’',  low,  subject,  where); 
if(  subject  >  high  )  printf("Number  too  HIGH  :  [%lf]  %lf  •  %*"n’',high,  subject,  where); 

} 

void  re4djConfigur4tion^le(C J*ARAMS  *config4>,  FJ’ARAMS  •fe4t^>,  int  cmdJine,  chnr  scrndjinroeQ) 

{ 

/•  This  function  is  specific  to  this  Fuzzy  Clustering  Progrnm.  It  rends  in  nil 
the  pnrnmeters  required  from  4  configurntion  file.  This  menns  the  progrnro  C4d 
be  run  in  multiple  formnts  nnd  reconfigured  ensily. 

Nenle  Prescott  OCT  93 

•/ 

FILE  econfigJn; 

check jnpttt.Arguments(cmdJine,  2);  /•  mnke  sure  the  config  file  wns  on  the  coramnnd-line  *f 

conflgJn  a  open.fiile j'e4d(cmd j>4me[l]);  /*  open  the  configurntion  file 

fgets(config.4>-«comment,  80,  config  jn);  /•  rend  comment  out  of  config  file  •/ 

f*c4nf(configin,”9(s**,  config.4>-^d4t4Ji4me);  /•  tend  dntn  in  file  nnme 

fsc4nf(configJn,  ’*9(s*',  config4)-.»initi4l^luster);  /«  rend  cluster  in  file  nnme  *f 
fscnttf(configJn,  '’9(s'’,  config4>-*cluster  Jinme);  /«  rend  cluster  out  file  nnme  s/ 

fscnttf(configJn,  '*9(s”,  config^)  — oerform-nnme);  /«  rend  perform  out  file  nnme  «/ 
fscnnf(configjn,  "His”,  drconfig^-«FKM jor.UFP  ); 
fscnnf(configJn,  ’'9Cd9Sd9(d9(dXlf9(lf9(d’',  drconflg^-*numberjirectors, 
drconfig^-*fentures4>er. vector, 
decenfig^><^initinl^lu*ter*, 
drconfig^— *finnljclusters, 

&config.4>-»fuzziness, 

8econfig4>*^stopwcriterin, 

8cconfig4>>*mnxJterntions  ); 

if(  (config-p-^FKMjor-UPP  ^  ’F’)  iete  (config4>-*FKM^r.UFP  ^  »U’)  ) 

{ 
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priBtfCCOHFIQ  ERROR:  P1«4M  tpacify  Fussy  k-meMs  [P]  or  UPP-ONC  [U]‘*b*'); 
priBtIC'Fussy  k-moBBS  U  bow  BMumsd'^a"); 
conftf-p**PKMj>r-UPP  >  »P*j 

) 

iBtjcks€kjrBBCs(coBfig^-^Bttmb«T.v«ctors,  2,  60000,  "Resd-CONPIGflJ'*); 
iBtJCksckJ4a(«(coBfi(^-*fe4tur«s^«rjvsctor,  1,  200.  ”Re4d-CONPlG(2]”); 

iat^ksckj’4Bfe(coBflf^'-«iBiti4l^lust€rs,  2,  64.  "R«4d*CONPIQ(3]**); 

iBl^hsckjr4Bg«(ceBlIg4>-»fla4liXla«ter«,  2,  64.  ''R64d-COMPlG[4)")^ 

iBt.xk«ck  J40(«(coBfi(.^f»*-*m4xJter4tioo*,  500,  3000,  "Raad-CONPIGlS]**); 

doobtexheck  j4age(config^^fusaioa«a,  1.0,  3.0,  ”R€4d*CONPlG{6]**); 

doubUxheckjr4a(e(eonfi|..p-^«topxriteri4,  1E»6,  1,  **Rc4d-CONFIG(7]")i 

f«4t.4>-^Bum-vcU  w  coBfig-p-«Bumb«r.vactor«i 
fe4t4»'-*Bttm^trt  a  coafig^<-«f«4tur««-p«r.vector; 
feat^^Bumxentroidt  a  coafig^>«iBiti4lxluac«ra; 

} 

void  s«roxD4trix(flo4k  ••smtx,  iat  row*,  iat  cola) 

{  /4>  Thia  fuBctioa  flUa  4  matrix  of  flosta  with  soroa.  a/ 
iat  i,  j; 

for(i  a  1;  i  <  rowa;  i++) 
for(j  a  Ij  j  <  cola;  i++) 
smtxfi][i]  a  0.0; 

} 

void  s«ro.vactor(flo4t  osvct,  iat  cola) 

{  /•  Thia  fuactioa  fitla  4  vector  of  floata  with  xoroa.  V 

ia»  j: 

ior(i  «!',)<  cola;  j+'f)  avct(j)  w  0.0; 

} 

void  soroxlm4trix( double  ••smtx,  iat  rowa,  iat  cola) 

(  /•  Thia  fuactioa  filla  a  matrix  of  doublea  with  seroa.  */ 

iat  i,  j; 

for(i  tt  1;  i  <  rowa;  i++) 
for(j  =  1;  j  <  cola;  j++) 

smtx(i]0]  a  0.0; 

} 

void  serowdvector(double  •svct,  iat  cola) 

{  /•  Thia  fuactioa  fill*  a  vector  of  doublea  with  seroa.  a/ 
iat  j; 

for(j  as  1;  j  <  cola;  )++ )  avct[)]  w  0.0; 

> 

void  makexnatriceaCC J’ARAMS  •coofig4),  M-PARAMS  •matrix.p,  SJ’ARAMS  •atata-p,  P J’ARAMS  •perform4>) 

{ 

/•  Thia  fuactioa  uaea  Numerical  Recipea  in  C  to  make 
a  number  of  matrices  for  thia  experiment.  Thia  ia 
for  eaae  in  reading  the  code.  All  matrices  are  initialiaed 
to  sero.  It  ia  aot  a  general  routine. 

Neale  Preacott  OCT  93 
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matrix^>«^feat-mtx  s  dmatrix(l,  conflg^-*number.vectora,  1,  config^-»featurea^er-vector); 

matrix.p'^mahipaxntx  =  dmatrix(l,  conflg^>*number_ve€tora,  1,  config^->*flnalx]ustera); 
matrix4>-^old-memberxntx  s  dmatrix(l,  config^}— •number.vectors,  1,  config^’-^finalxluatera); 

matrix.^)— •centroidxatx  s  dmatrix(l,  config.p>«finalxluatera,  1,  config^-»feature8^>er.vector); 
matrix^^diat xntx  s  dmatrix(l,  config.p-*number.vectora,  1,  config^-*^finalxlu8tera); 

matrix^— ^Fxovarxntx  s  dmatrix(l,  conflg.4>^featores-per.vec}or,  1.  config^-»feature8.per.vector); 

seroJlmatrix(matri.x.^>'>^featxILtx.  coafig^— •number .vectors,  config^?— •(eatttrea4)er  jvector); 
sero jdmatrix( mat rix.^-^mabipsxntx,  config^)— •number.vectora,  config^-^fiaalxluatera); 
serojdmatrix(matrix^-*old-member  jntx,  config-p-»numberjvectora,  config^>-»finalxluaters); 
xeroulmatrix(matrix^-«centroid  jntx,  config^)— •finalxlusters,  config4>-^»feature8^er.vector); 
aerojdmatrix(matrix4>-»diatxntx,  config-p-*number_vectora,  coafig^>-*finalxluatera); 
aerojdmakrix(makrix^-*Fxovarxntx,  config4>^features^>er .vector,  config^-^featurea^er.vector); 

atata^>^mean.vct  ■  dvector(l,  config.4>-«features^erjirector); 
atats.p>«var.vct  a  dvector(l,  config.4>-<*featurea4>er^ector); 

serojdvector(atata^>-»mean.vct,  config.p-^featurea^er .vector); 
sero jdvector(atata.p<-»var .vet,  config.4>«^featurea.4icr.vector); 

perform.4>-*Fus3.£V  a  dvector(l,  config^«*finalxluatera); 

perform.4>-»*Av.Denaity  a  dvector(2,  coofig.p^6aa]xlaatera); 
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p^rform^^Pu-tition  a.  dv«ctor(l,  coaflg4)-*fia4lxltt«ter»)i 

a«ro^v«ctor(p«rform^-»PuBiJiV,  config^-^finaljclusUrs); 
sero^v«ctor(p4rforBi^-*AvJD«n«ity,  coBfig^*-*fia4l^lu»tc»); 

B«rojclv«ctor(p«rform^-^Partitioa^«n»ity,  coafig^-»fiB4l^ltt«ter«); 

/  V 

void  dol«t«^4trice«(CJ*ARAMS  vcoafig^i,  MJ^ARAMS  •m*tnx4»,  S-PARAMS  P J’ARAMS  «perform^) 

{ 

/v  This  fuaction  u*e«  Nuoaoric*!  Recipe*  ia  C  to  free 

4  Bumber  of  m4trice«.  Thi*  i*  for  eaee  ia  readiag  the  code. 

It  U  aot  4  geoer4l  routine,  i.e  Specific  to  thi«  code. 

Neele  Preecott  OCT  93 
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freejdm4tnx(m4trix^>-^fe4t jcntx,  1,  config4>-^aumber.vector«,  1.  config^<^fe4ture*4>er.vector); 

free^in4trix(m4trix^-*m*hip*.^tx,  1,  config4>— •numberjvector*,  1,  coafig_p-*fin4]^lutters)i 

freejdm4trix(in4trix^— •old.jnember-mtx,  1,  coafig^)'>*nuaiber.vectore,  1,  coBfig^-»fiB4l.xlu*ter«); 
freej(lm4trix(m4trix^>-*centroiduBtx,  1,  coafig.4t*>*fin4l.xltt«t4r«,  1,  coBfig^-*fe4ture*.4>er  .vector); 
free^m4trix( matrix.^ -^diitjaitx,  1,  config.p-*aumberjvector«.  1,  coBfig4>-*fin4l^lu«ters); 

freejlm4trix(m4trix^-*Fjcov4r.Atx,  1,  coafig^-*fe4tttfe«4»erjvector»  1.  coBfig.p-»fe4ture«^er_vector); 

freeulvector(*t4t«.4)-^me4a.vct,  1,  config.p^fe4ture«4>er.vector): 
free^vector(*t4te.p^v4r.vct,  1,  config.p<^fe4ture*.4ier-vector); 

free^vector(perform4>>*Pu2x.HV,  1 ,  ceafig4>*^fin4l^itteter*); 

freejlvector(perform4»-»Av.Deniity,  1,  C0Bfig4>*^fiB4l^lttetere); 

free^vector(perforin^^P4rtitioa.Den*ity,  1.  coafig.p-»fia4lj:iu*tere); 

} 

void  getjd4t4.«t4U(C.PARAMS  •coafig^.p,  MJ*ARAMS  emetrix.^).  SJ^ARAMS  ••teu.p) 

{ 

/*  Thi«  function  celcuate*  the  mean  end  verience  of  the  date  aet. 

Which  i*  in  fact  the  initial  centroid. 

The  function  u«e*  Numerical  Recipe*  in  C. 

The  input  i*  a  matrix  of  double*  (column*  being  feature*,  row  are  feature.vector*). 

The  output*  are  two  vector*  *  mean  and  variance. 

N. Prescott  OCT  93 
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int  col,  row,  numjiero; 

float  average,  avjdev,  atdjdev,  var,  *kew,  hurt,  etmp.vct; 

tmp-vct  s  vector(l,  config.p>-*number.v«ctor*); 

for(col  s  1;  col  <  config^)-»feature*4>er. vector;  col4’>(') 

{ 

num  jero  s  0; 

for(row  s  i;  row  <  config^)'-*number.vector8;  row+'l>) 

{ 

tmp.vct[row]  s  (float)  matrix.p-»feat jntx{row][col}; 
if(tmp.vct[row]  ss  0.0) 

numjero-f  •b; 

} 

if(  numjiero  =se  config.p-*number.vectors) 

{ 

average  s  0.0;  /e  Thi*  avoid*  an  NRC  error  when  the  vecto.  o  V 

var  wt  0.0; 

} 

el*e 

moment(  tmp.vct,  config^i-^number.veclor*,  ^average,  &avudev,  dr*tdjdev,  drvar,  3£*kew,  &kurt); 
*tat*.^-*mean.vct[col]  s  (double)  average; 

*tat*.P'^var.vct[col]  s  (double)  var; 

} 

free.vector(  tmp.vct,!,  config.p-^number.vectore); 

} 

void  load Jeatore*(CJ’ARAMS  econfig^,  MJ*ARAMS  ematrix^)) 

/•  This  function  reads  in  data  from  an  ASCII  file  and  place*  it  into 

'’feature jnatrix".  The  number  of  feature.vector*  and  features^er.vector 
must  be  specified  a*  the  data  file  must  not  have  a  header. 

Neale  Prescott  OCT  93 
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int  row,  col; 

FILE  edataJn; 
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d4t4jii  ■  op«B^l«jr44d(confif^-*d4t4^4m«); 
for(row  ■  1;  row  <  coaflf4>^auinb«rjir4ctor«i  row-f -f ) 
for(ce)  m  1;  col  ^  coDflf^^f«4tur«i^erjifector;  col4>4^) 
Uc4af(d4t4jn,  **9(11**,  iem4trix.,p-^l«4l  jntx[row](col]  ): 

f€lOM(d4l4iB)i 

} 


*/ 


void  lo4d^«iitroid«(CJ* ARAMS  •coitfig4»,  M  J^ARAMS  •motrix-p,  double  eecov.^,  double  edet^*,  double  •oprior^) 

{ 

/•  Thi«  fuaction  read*  ia  data  from  aa  ASCII  file  aad  place*  it  iato 
"ceatroidjnatrix**.  The  number  of  centroid*  and  feature*^er^eatroid 
must  be  fpecified  a*  the  data  file  must  not  have  a  header. 

Neale  Prescott  08NOV  93 
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iat  i,  row,  col,  cntS; 

FILE  edataJni 
char  tmp^tring[90]; 


dataJn  a  opeB.^lejead(config4»-^cluster^ame); 
cats  s  1; 

for(i  a  1;  i  <  coofig4^<^flnalxlttsters;  i+  +  ) 

{ 

fgets(tmp^tringi  60,  dataJn); 
fscanf(datajn,  ’^KlP,  draprior^[i]  ); 
fscanf(dataJa,  "Klf,  irdet4>(i]  ); 

for(row  a  1;  row  ^  conflg^>«feature*^>er jvector;  row>f +»  cDt2*l>*f ) 
for(col  a  1;  col  <  coBflg^««feaCttres4>er.vector;  0014*4) 
fscaBf(dataJn,  ''9(ir',  dECOv.4>[cnt2][col]  ); 
ffets(tmp^triBg,  60,  dataJn); 

} 

fgets(tmp^tring,  60,  dataJa); 

for(row  a  1;  row  <  coafig^^-^finalxlusters;  row44) 

for(eol  a  1;  col  <  conflg^-*features4>er.vector;  eol44) 
fscaBf(dataJn,  ’'Idf*,  6cmatrix4>«*centroidjntx(row](colj  ); 

fclo*e(dataia)‘, 

/  •i 

void  performJJFP.ONC(C^ARAMS  ecoafig^,  M^ARAMS  ematrix^,  FJ*ARAMS  efeat^^,  S-PARAMS  Mtat^,  P  J’ARAMS  •perform4>) 

{ 

/«  Perform  the  unsupervised  Fussy  PartitioaoOptimal  number  of  classes 
routine.  See  Reference  1,  listed  in  the  header  comments. 

The  "fussiness”  and  "stopwepsilon”  are  adjustable  parameters 
NOTE  "fussiness  >  1”  and  ”stop.^psilon  [0,l]" 
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int  cluster; 


initiali*ejclusters(config.4),  matrix^,  stat.^); 


for(  cluster  a  conflg^-^initial^lusters;  cluster  <  config^— •finaljclusters;  cluster-f  4 ) 

{ 

if(PRINT JDIAG)  printf(””n”nCentroid*  s  %d”n”n”,  cluster); 
feat4>-^num-centroids  a  cluster; 


calc  JussyJc^eans(config4?,  matrix^,  feat^); 
if(PRlNTJDIAQ)  aave^entroid*( matrix^,  feat^i); 
calcJnssy-mle(config4>,  matrix4>,  feat^,  perform^!); 
save^entroids(matrix.,p,  feat^); 

} 

if(PRINTJDIAQ)  store4>erformancejlata(config-p,  perform^)); 

/  */ 

void  initialisejclusters(C^ARAMS  econfig^k,  MJPARAMS  «matrix4>,  S-PARAMS  estats^)) 

{ 

/•  Refer  to  pp775.  Section  2.B  of  the  UFP-ONC  paper  for  why 
the  2nd  centroid  is  chosen  this  way. 

Initial  cluster  centre  is  at  the  data  mean. 


CoaMCtttiv«  c«alr«i  /*)  R»m4om  aoioudU  of  Stondord  Doviotioa  from  meon 

N«U«  Proocott  30JAN94. 

V 

iol  cluttor,  foature,  p«eudo,  i; 
dottblo  mnU; 

•raBd(l)i 

for(cluot«r  •  1;  clutter  <  conflf^-^flnal^iuttert;  clu«ter4>'f ) 

{ 

i((  clu>t*r  «  1) 

for(fe4ture  m  l;  feature  <  conftg4»^feature4^er.vector.  feature^^) 
matrix^r-^centroid  Jtntx(clu«ter](feature]  m  ttatt^-^meaa.vctlfeature]; 

elte 

for(ial;  i  ^  10;  i4-4‘) 

{ 

pceudo  m  (raad()  %  conflg^r^number.vectore)  1; 

for(feature  m  1;  feature  <  eonfig^-^featuret^er.vector;  feature44) 

matrix^^centroid  Atx[clueter][(eature]  -f  s  matrix^-^feat  Atx[p«eudo][feature]/10.0; 

} 


} 

} 

void  calcJ'ttaayJc^eaiit(C^ARAMS  «config4>,  M^ARAMS  ematrix-p,  P-PARAMS  efeat^) 

{ 

/«  Refer  to  pp774,  Section  3. A  of  the  UPP*OKC  paper  eqnt  1.3,3. 4,5 
See  Reference  3 
Neale  Prescott  OCT  93. 

V 

int  iteration*  m  0; 
double  objectiveJn; 

calculate^dUt  J^KM(config^,  matrix^,  feat4>); 
comptttejuemberJRKM(config^,  matrix^),  feat^r); 
do 
( 

compute.xentroidt  JKM(conflg4>,  matrix^p.  feat^)); 
copy.jnemberJo.OLO(matrix4>,  feat4>); 
calcttlate^iit JPKM(conflg^,  matrix^),  feat.p); 
computejnember  JKM(config4>,  matrix^),  feat^>); 
objectiveJn  m  compute^bjective  JKM(matrix.p,  feat4>); 
iteratioDe4>^; 

if(PRINX.DIAQ)  printf('*FKM  Iterations  :  9(4d  :  Obj  s  9(If”ii",  iterations,  objective.Jn); 
if(  iterations  ss  config^— 'max Jterations) 

priatf(”FKM  •  maximum  iterations  [9(d]  reached.  Stopping. *'n’'.  iterations  ); 

) 

while(  (objectiveJn  >  config^>'-*stop.£riteria)  SeSe  (iterations  <  conlig^>-*roaxJterationB)  ); 

} 

void  copyjnemberUoJ:>LD(M  J’ARAMS  ematrix^),  F^ARAMS  efeat^)) 

{ 

/•  30  OCT  93  :  The  tricky  use  of  pointer*  has  replaced  the  element  by 
element  copying. 

NOTE  :  That  the  memberships  matrix  will  contain  old  data, 
care  must  be  taken  not  to  use  these. 

Neale  Prescott  OCT  93 

v 

double  eetmp4>tr; 

tmp^tr  «  matrix4><>«old.^emberjntx; 
matrix^*«old.Aember jntx  s  matrix^— >mship*^tx; 
matrix.4>«»mships.jntx  »  tmp.^tr: 

) 

void  colcuUte^iit J'KM(CJ’ARAMS  •config.p,  M-PARAMS  •raatrix4>,  FJ’ARAMS  vfeat^)) 

{ 

/•  This  function  computes  the  distance  from  every  sample  to  each  of  the 
centroid*.  The  value*  are  stored  in  the  matrix  "dist  jntx**. 

SQR(a)  is  defined  in  nrutil,  Numerical  Recipes 
Neale  Prescott  OCT  93 

V 

double  etmpjfst,  dist; 
int  row,  cl,  ft; 
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tmp^At  m  dv«ctor(l,  coBfif4>««f«4tur«t^«r jir«ctor)i 
for(row  m  1;  row  ^  («4t^^aum.vcUi  row'f -f) 

(or(cI  ■  1;  cl  ^  fe4t^-*nuinx«Btrot4s;  cl-f^) 

{ 

for(ft  ■  1;  ft  ^  f«4t^-^nuin^tr«;  ft'f'f) 

tmpulst[ft]  m  m4trix^<-»fe4k jnlx[row][ft]  —  motrix^— >ceatroid^tx(cl][ft]; 
dUt  s  0.0; 

for(ft  s  1;  ft  <  f«4t.p'-*num^tr«;  fk*f *f) 
di*t  4*  DSQR(tmpjdtk{ft])i 

m4trix.^^di*t.jntx[row]{cl]  *  diet; 

) 

fr«4jdv«ctor(tmp^«t,  1,  f€4t^-*num Jtrx); 

}  ^ 

void  comput«JiieiDberJ^KM(CJPARAMS  «config^,  MJ^ARAMS  •motrix.^,  F.PARAMS  sfeat^) 

{ 

/•  Thi«  function  compute*  the  membership  of  every  d4t4  sample  to  each 
of  the  clusters  (classes).  See  Eqn.2  of  Reference  1. 

Neale  Prescott  OCT  93,  FEB  94 

V 

int  row,  cl,  FLAG; 
double  *um.^st; 

for(row  m  1;  row  <  feat.^)— »num.vcts;  row4>4>) 

{ 

sum^st  a  0.0; 

FLAG  *  -1; 

for(cl  a  1;  cl  <  feat^--*num-centroids;  cl-f-f ) 

{ 

if(matrix.^--*dist.antx[row][cl]  ^  0.0) 
sum^st  +s  (1.0/matrix^-»dist.jntx[row][cl}); 

else 

FLAG  a  cl; 

} 

for(cl  s  1;  cl  <  feat.^-*num.xentrotds;  cl4’+) 

{ 

if(FLAG  S3  -1) 

matrtx^-^mships^tx(row][cl]  s  (1.0/inatrix-p'-*dist  jntx(row]{cl])/sum^st; 
else 

if(cl  3=  FLAG) 

matrix.4>^m*hip*.jntx(row][cl]  s  1.0;  /•  This  point  is  at  the  centroid.  Membership  must  be  1 

else 

matrix^-*mships-mtx(row][cl]  s  0.0;  /e  Must  have  zero  membership  in  all  other  clusters  •/ 

} 

> 

>  ^ 

void  computej:entroid*  JKM(C.PARAMS  econfig^),  MJ’ARAMS  *roatrix-p,  F-PARAMS  *leat^) 

{ 

/«  Computes  the  new  centroids  of  the  Fuzzy  K  Means  as  per  eqa.3  of 
Reference.! 

DANGER  :  If  the  denominator  is  zero  this  will  explode.  But  it  should 
be  ’’impossible”  for  this  to  occur! 

Note  :  ”eta”  is  a  cluster  validity  check  from  a  paper  by  Krisnapuram  and  Keller 
IEEE  I^ans  on  Fuzzy  Systems  Vol.l,  No. 3,  May  1993. 

Neale  Prescott  OCT  93 

«/ 


int  row,  cl,  ft; 

double  enumerator; 

double  tmpjnemb,  denominator,  eta; 

numerator  s  dvector(l,  feat4>^numJtrs); 
for(cl  s  1;  cl  <  feat^— »numjcentroids:  cl44) 

{ 

zero..ilvector(numerator,  feat^— •numJtrs); 
denominator  a  0.0; 
eta  3  0.0; 

for(row  3  1;  row  <  feat^>>*num.vcts:  row4>4) 

{ 
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tmpjnemb  m  DSQR(m»lrix-p-^m«hip«^tx[row][cl)); 

6tA  (trapjnemb  •  mft(rix^-~*dUt^tx[row]{cl])i 

for(ft  s  ft  <  fext^->-*numJtMi  ft-f-f ) 

aum«r»tor[ft]  +s  tmpunemb  •  mxirix^-^fext jntx[row)[ft]; 
deaominfttor  tmpjnemb; 

} 

for(ft  s  1;  ft  <  fe«t4>^numJtrt;  ft-f  4) 

mxtrix^-^centroid  jntx(cl](ft]  s  nuin«rator[ft]/dBnomiaxtor; 
et«  m  et&  /  denomtnatori 

} 

fr««uivector( numerator,  1,  fe4t^*«num Jtr*}^ 

/  V 

double  compute^bjcctiveJKM(M^ARAMS  •matrix^),  PJ’ARAMS  efeat^) 

{ 

/*  See  eqB.4  of  Reference  1. 

DMAX(a,b)  U  in  nrutil,  Numerical  Recipee) 

Neale  Preecott  OCT  93 

V 

double  maximum  s  0.0,  difference; 
ittt  row,  cl; 

for(row  s  1;  row  <  feat^t-^num.vcCc;  row44) 
for(cl  w  1;  cl  <  feat^— »num^entroid4;  cl44) 

{ 

difference  w  fab«(  iaatrix.4>^m«hip«-mtx[row]{cl]  —  matrix^>»old-member jntx[row][cl]); 
maximum  s  DMAX( maximum,  difference); 

} 

return( maximum); 

} 

void  print jnember«hip«(  MJ^ARAMS  «matrix4>,  F^ARAMS  efeat-p) 

{ 

/•  Thii  function  print*  out  the  member«hip*  for  each  data  point 
and  i*  primarily  for  de*bugging.  However  it  could  be  quickly 
modified  for  u*e  in  Qnuplot. 

Neale  Preecott  OCT  93 

•/ 

int  i,  j; 

printf(*’#  Membership*  for  [9(d]  centroid*  ‘*n”,  feat4>-»ouraxentroids); 
for(i  s  1;  i  <  feat^^num.vct*;  i44) 

{ 

for(  )»!;}<  feat^^num.xentroid*;  j44) 
printf("%lf  ",  matrix^-*m*hip« jntx[i][j]  ); 
printf("  "n"); 

) 

} 

void  iave_centroids(M J’ARAMS  ematrix^,  PJ*ARAMS  efeat^) 

{ 

/•  The  final  cluster  coordinates  must  be  saved.  This  is  because  the  UPP-ONC 
algorithm  starts  with  two  centroid*  then  increases  up  to  the  MAX.CLUST. 

Once  the  optimum  number  of  classes  is  found  we  would  like  to  locate 
the  cluster  coordinates  without  re-calculating  them. 

NOTE  :  Modify  this  to  send  it  to  the  file  specified  in  the  configuration  file 
Neale  Prescott  OCT  93 

«/ 


int  cl,  ft; 

printf(’*#  Cluster  locations  for  [%d]  centroids  of  [96d]  dimensions  :  “n",  feat^-»nurojcentroids,  feat4>-»num Jtrs); 
for(cl  s  1;  cl  <  feat^— *num^entroids;  cl44) 

{ 

for(  ft  s  1;  ft  <  feat^-»numJtrs;  ft44) 

printf("9Se  ",  matrix^-*centroid-mtxfcIj[ft]  ); 
printf("“n"); 

} 

} 

void  save^i*tances(C J’ARAMS  econfig-p,  MJ^ARAMS  smatrix^) 

{ 
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/•  For  dcbttfftag  purpo««« 

N«4l«  Prascotl  OCT  93 

V 

int  row,  cl; 

priatf(" Distances  from  ceatoids^ii'*); 

for(row  m  1;  row  <  config^>*aumber-vectars;  row>f4) 

{ 

for(cl  w  1;  cl  <  coiifig^*^fin4l^lustersi  cl4‘  +  ) 
printf('*%lf  matrix^*^dist^tz[row][cl]); 

printf(**“n’'); 

} 

/  v 

void  calcJu»yjnle(C J*ARAMS  econfig^),  M^ARAMS  •matrix^,  PJ’ARAMS  efeat^,  P-PARAMS  eperform^)) 

{ 

/*  This  function  performs  the  Puaay  Maximum  Likelihood  Estimate 
as  per  Reference  1. 

Note  :  Due  to  the  siae  and  number  of  Puaay  Covariance  matrices 
this  function  works  on  one  cluster  (centroid)  at  a  time  to 
limit  the  amount  of  memory  required. 

Neale  Prescott  OCT  93 

V 

double  objective^n; 
int  iterations  a  0, 
cluster,!  j; 

initialise..FMLE.membs(config^,  matrix^,  feat^); 
set^uiayxov  Jdentity(matrix4>,  feat.4>); 

for( cluster  s  1;  cluster  <  feat^-^numucentroids;  cluster++) 

{ 

calc^entroidJ*rob( matrix^,  feat^,  cluster); 
calcjdist^MLE(matrix.p,  feat^t,  cluster,  perform^); 

) 

computeunember-PKM(config.4>,  matrix-p,  feat^); 

do 

{ 

compute^entroids  JKM(config4>,  matrix^),  feat.q>); 
for(cluster  si;  cluster  <  feat^^numwcentroids;  cluster+4) 

{ 

calcJF'a:ovar(matrix.p,  feat^,  cluster); 
calc^entroid J’ro^matrix-p,  feat^,  cluster); 
calc^ist^ML£( matrix^,  feat^,  cluster,  perform^;)); 

) 

copy jnember Jo-OLD(matrix-p,  feat^); 
computejnember  J'KM(conflg^,  matrix^),  feat^); 
objcctiveJn  s  compute^bjective JKM(matrix^,  feat^); 

iterations 

if(  iterations  ss  coQfig4)^max Jterations) 

printf(*’FMLE  •  maximum  iterations  [9(d]  reached.  Stopping. iterations  ); 
if(PRINT J^IAG)  printf(”FMLE  :  [9(4d]  Obj:  KlP^n",  iterations,  objectiveJn); 

} 

while(  (objectiveJn  >  config^'^stop^riteria  )  &&  (iterations  <  config-p-»max Jterations)  ); 
calcxlustJ*erform( matrix^,  perform^,  (eat4>); 

} 

/.  =  =  =  »  =  =  =  =  „  =  =  =  =  =  ^  =  =  =  =  »  =  =  .  =  =  =  =  =  =  =  =  =  =  =  =  =  =  =  ^ 

void  initialise J'MLEjnembs(C.PARAMS  «config4»,  M^ARAMS  ematrix^,  F-PARAMS  efeat^)) 

/«  This  function  initialises  the  memberships  to  (1/nura  Of  cluster)  for 
the  first  iteration  of  the  FMLB.  See  Gustafson^Kessel,  or 
"Fitting  an  Unknown  Number  of  Lines  and  Planes  to  Image  Data  through 
Compatible  Cluster  Merging"  by  R.  Krishnapuram  and  C-B  Freg,  Pattern  Recognition, 

Vol.25,  pp.  385.400,  1992. 

Neale  Prescott  Feb  94 

V 

int  i,  j; 

double  MEMB; 

MEMB  s  1.0/((double)feat^— ^numucentroids); 
for(i  s  1;  i  <  config4>-»number.vectors;  t++) 
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(or(i  s  1;  j  ^  (e4t-p-*nttm^«nkroid«;  i+*f ) 
mftthx4>>«m»liip«^tx[i]{j]  a  UEMB; 

)  _  _  ^ 

void  tot  JuBxyxov  jdenkity(M-PARAMS  •mxtrix^,  F^ARAMS  vtoot^) 

{ 

/•  Thi*  (uaction  iaiUxli»««  the  Fuaxy  Covanxnce  Matrix  to  the  Identity  matrix  for 
the  firet  iteration  of  the  PMLE.  See  Guetafeon-Keeael,  or 
"Fittiof  an  Unknown  Number  of  Line*  and  Plane*  to  Image  Data  through 
Compatible  Clucter  Merging”  by  R.  Krithnapuram  and  C-H  P^eg,  Pattern  Recognition, 

Vot.25,  pp.  385-400,  1992. 

Neale  Preecott  Feb  94 

V 

int  1,  y, 

for(i  s  1;  i  <  feat^t-^numJtra;  i4-4-) 
for(i  3  1;  j  <  feat-p-^num^tre;  j  +  'f) 

if(  i  ^  j) 

matrix^-*P^ovar^tz[i][j]  a  0.0; 
eUe 

matrix.^>-^F^ovar.antx[i][j]  a  1.0; 
matrix.4>-»P^et  a  1.0; 

) 

void  calc..xeBtroidJ*reb(M  J^ARAMS  ematrix^,  F.PARAMS  efeat-p,  int  clu«ter) 

{ 

/•  This  unction  calculate*  the  a  priori  probability  of  *electtng  the  ’M^th  cUuter 
Refer  tp  eqn(8),  pp774,  of  Reference  1. 

Neale  Preecott  OCT  93 

V 

int  i; 

double  PJ  a  0.0; 

for(i  a  1;  i  ^  feat^}-*num.vct*;  i+4’) 

Pi  -fa  matrix4)-«m*hip*jmtx[i][cluster]; 
matrix4»-*aPRI0RI  a  Pi  /  ((double)feat4»-»aum.vcts); 

}  _  _  __  _  ^ 

void  calc J'xovar(M.J*ARAMS  ematrix.^),  FJ*ARAMS  efea^^),  int  clu*ter) 

{ 

/e  Thi*  function  calculate*  the  Fuaay  Covariance  Matrix  foi  the  specified  clueter. 

Thi*  i*  *hown  at  pp.775,  eqn(9)  of  Reference  1. 

The  member*hip  matrix  i*  now  u*ed  to  hold  the  poateriori  probabiltitie*,  fa(i|Xj)  (see  pp774) 

Numerical  Recipe*  in  C  are  u*ed  to  find  the  inver*e  and  determinant  P*(-l)  and  )F| 

Neale  Pre*cott  OCT  93 


double  eXsubV,  eetinp^,  denom,  ew,  eev,  wmax,  wmin,  trap^et,  d; 
int  row,  ft,  i,  j,  k; 

X*ubV  a  dvector(l,  feat.4>— •numitrs);  /•  vector  from  sample  to  current  centroid  «/ 

tmpJ  3  dmatrix(l,  feat^-»numitrs,  1,  feat-p^numJtrs  ); 
w  3  dvector(l,  feat^-*numitr*);  /«  SVD  requirement  «/ 

V  3  dmatrix(l,  feat4>-*numitrs,  1,  feat4»— »numitr*  );  /•  SVD  requirement  «/ 

zerojdvector(XsubV,  feat^— *numitr«); 
zero.jlmatrix(tmp^,  feat^-^numitr*,  feat4)-^numitr*); 

/e  calc  the  Fuaay  covariance  matrix  for  current  cluster  */ 

/• . y 

denom  a  0.0; 

for(row  3  1;  row  <  feat^— »num.vct*;  row-h-f) 

{ 

xerojdvector(X*ubV,  feat..4>-^numitr*); 
for(ft  3  1;  ft  <  feat4>-*numitr*:  ft^-f) 

X*ubV(ft]  3  matrix^— »feat.jntx[row][ft]  matrix^-*rentroid Jntx[clu8ter]{ft]; 

for(i  3  1;  i  <  feat4>-*numJtrs;  i*f +  ) 
for(j  3  1;  i  <  feat^>^numitr*;  j++) 

tmpi^[i](i]  +*  (X*ubV[i]eX*ubV[j]  •  matrix^— »m*hip8jntx[row][cluster]); 
denom  matrix^-^mshipsjntx[row][clu*ter]; 

} 
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for(i  «  1;  i  <  i-f  +  ) 

for(j  a  li  j  ^  f«4l^-*oum Jtrs;  i4“f ) 
Imp  m  Imp 


/•  D«lerimo4Qt  4nd  Inverse  C4lcul4tion  u«ing  SinguUr  V4lue  Decompo«iti  >0  V 

^  .  •/ 

/«  Decompose  PuTtsy  cov4n4nce  m4lrix  into  (2)  orlhonorai4l  xnd  4  di4gon4l  in4lnx  ^ 
d«vdcmp(  tmpJ*.  fe4t4>~*iium Jtr*.  fe4t4>-»nttmJtr<,  w,  v  ); 

/•  Tr4n»po*«  the  U  m4tnx  which  W4«  returned  in  Imp  J  V 
for(i  w  li  i  ^  fe4t^— »numJtrsi  i*f4-) 
for(j  B  i;  j  <  fe4t^<^numJtrei  j-f  +  ) 

i<(  i  ^  i)” 

d  B  tmpj*[i]y]; 
tmpji‘[i][i]  B  tmpJ'[j][i]; 
tmp^[j)[ij  s  d; 

> 

/«  Multiply  the  inveree  dixgonel  (1/w)  by  tr4n«po«e  U  «/ 
for(i  a  1;  i  <  fe4t4»>»num Jtre;  i+  +  ) 
for(j  B  1;  j  <  fe4t^— knumJlre;  j-f  +  ) 
tmpj'[i][j]  =  tmpJ'[i][j]  /  w(i]; 

/•  Perform  fin4l  motrix  cxlculxtion  to  find  Inveree  of  Pusay  Covotixnce  «/ 
xerojdin4trix(m4trix^— •F^ovxr^tx,  tc«t^>*numJtrf,  fext^-^num Jtre); 
for(i  s  1;  i  <  fe4t^-»nuin-ftr«;  i  +  - 
for(j  B  1;  j  <  fe4t^-*num_ftr«;  j*f  +  ) 
for(k  B  1;  k  <  fext^— •numJtre;  k-f4>) 

m4t'‘ix-p-*F-£0V4r^tx[i][i]  +b  (v[i][k]  «  tmpj[kj[i}); 

/«  Determine  the  CONDITION  number  of  the  SVD  </ 
wmxx  s  0.0; 

for(jBf;  j  <  fe4t.4>«»numJftr«;  ]++) 
if(wp]  >  wmxx) 
wmxx  a  wQ]; 

wmin  B  1.0e20; 

for(j  B  1;  j  <  fext4>-*numJtr«;  j'f  +  ) 
if(w[j]  <  wmin)  wmin  b  w[j]; 

/•  Refer  to  Numericxl  Recipee  in  C.  Bxeicxlly  indicxte*  nexr  eingulxr  roxtrix  •/ 

/•  for(j  B  1;  j  <  fext^><>nuro Jtre;  j-^'f ) 
if(w[j]  <  (wmxx  •  ].0e>6))  w[j]  s  0.0; 

V 

if(PRINT.^IAG)  printf("Condition  No.  :  9(if“o’\  wmxx/wmin  ); 


/•  Cxlculxte  determinxnt  of  Puzay  Covxrixnce  mxtrix 
/e  Note  :  detA  s  det(iovA)  «/ 

/•  The  dixgonxl  mxtrix  contxine  the  eigenvxlues  *f 

/«  The  product  of  the  eigenvxluee  i«  the  deteiminxnt  *f 

!•  See  Linexr  Algebrx  by  Strxng  1988  xnd  NRC  2nd  Ed.  V 


/•  Log*  hxve  been  ueed  to  reduce  riek  due  to  over/underflow  •/ 

mxtrix^-^F.jdet  b  0.0; 

for(j  =  1;  j  <  fext^>— ^numJlre;  1  +  +  ) 

>f(»'til  *  0.0) 

mxtrix.^— »P^et  -fs  logl0(  w[j]  ); 
mxtrix^-*P^et  b  pow(10.0,  mxtrix4>— 'Pjdet); 

/«  Delete  xll  temporxry  matrices  xnd  vectors 

free«dmxtrix(tmpJ^,  1,  fext-p— »num Jtrs,  1,  fext4>— *num^trs  ); 
free^vector(  w,  1,  fext-p— »num Jtrs); 

freejdmxtrix(v,  1,  fext^>^num^tTS,  1,  fext^-*num J^trs  ): 
free^vector(XsubV,  1,  fext^*^numJ^trs); 

) 

void  print.Pus2yxovxrixnce(M J^ARAMS  *inxtrix-p,  FJ*ARAMS  efext^) 

{ 

/•  Two  pupotet  ;  1.  Save  Fuzzy  COvariance  Matrix  i..r  use  in  the  Classifier 
2.  Debugging 
Neale  Prescott  OCT  93 

V 
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int  i.  j; 

priiitf("#  Inverted  Pussy  Covariance  Matrix  [9(d  X  %d|“ii*‘,  feat-p-^num Jtr»,  (eat^'^nuin Jir«); 
printf("9(e“n”,  matrix-p«^aPRlORI); 
pntttf("9(e“n’',  matrix^-«F^et); 
for(i  »  l;i  <  feat-p— num Jtr«;  i-f+) 

{ 

for(jBl;j <feat^>-»aum  Jtr«;i  -f +  ) 
printf('*9(e  matnx^-*Fjcovar.mtx[i][i])i 
printfC'  "n”); 

} 

}  ^ 

void  calc^iat  JMLE(M J*ARAMS  ematrix-p,  F-PARAMS  efeat^,  int  cluster,  P_PARAMS  *perform^) 

{ 

/«  This  function  calculates  the  distance  for  each  sample  to  the  current  cluster 
See  Reference  1,  pp774.  eqn(7). 

It  also  passes  partial  results  to  the  perfoimance  calc,  refer  eqns(12)(14)  on  pp775. 

Neale  Prescott  OCT  93 


double  eXsubV,  etmpR,  mult; 
double  dist; 
int  row,  i,  j; 

XsubV  B  dvector(l,  feat-p— •num^trs);  /•  vector  from  sample  to  current  centroid  */ 

trapR  =  dvector(l,  feat^— •num Jtrs); 
serojivector(XsubV,  feat^— »numjftrs); 
sero^vector(tmpR,  feat^>— »num^trs)-, 

for(row  =  1;  row  <  feat^-*num.vcts;  row4>  +  ) 

{ 

for(i  s  1;  i  <  feat4>-»numJtrs;  i  +  +  ) 

XsubVfi]  s  matris^'<*feat«jntx(row}[i]  —  matrix^*^centroidjnttx[clttster}[i]i 

for(i  =  1;  i  <  feat4>-*num4trs;  i+  +  ) 

{ 

tmpR(i]  =  00; 

for(j  =:!;)<  feat^^num Jtrs;  j  +  +  ) 

tmpR(i]  +»  (matrix4>^F^ovar^tx[i][i)*XsubV[i)); 

} 

dist  s  0.0; 

for(i  B  1;  i  <  feat^->*num Jtrs;  i*f  +  ) 
dist  B  dist  +  (XsubV[i]«tmpR[i]); 
if(dist  <  0.0) 

{ 

printf(”OVERFLOW  ;  NEGATIVE  FMLE  DIST  «If“n"ii’’,di»«); 

dist  s  0.0; 

} 

if(dist  <  1.0) 

{ 

perform^— »S1  +=  matrix^^m8hips-mtx[row][clusler], 

if(PRINT J)IAG)  printf(”Distance  is  inside  ellipsoid  9(e*'o'’.dist); 

} 

mult  s  (Iogl0(matrix_p'~*F^et))/2.0;  /•  Take  ”sqrt'’  hopefully  without  underflow  *j 
mult  s  (pow(10.0,  mult))/(natrix^)-»aPRIORI; 
dist  s  mult  •  exp(dist/20); 
matrix4>-~*dist-mtx[row][clu8ter]  s  dist; 

} 

free^vector(XsubV,  1,  feat^-«num Jtrs); 
free.jlvector(tmpR,  1,  feat^-»num  Jtrs); 

}  _  ^ 

void  calcjclust_Perform(M J’ARAMS  emalrix-p,  PJ’ARAMS  eperform.^,  FJ’ARAMS  •feat4>  ) 

{ 

/e  See  Krishnapuram  and  Freg,  Pattern  Recognition,  vol26  1992  or 
Besdek,  or  Bain.  And  Gath  and  Geva  in  Reference  1. 

Neale  Prescott  Feb  94 
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tnl  i,  j,  Cljcnt; 


perform^-^Fun JIV[fe4t^>«nuin.xentroi<l«]  s  0.0; 
perform^)— »S2  as  0.0; 

perform.4>-«Av JD«a«ity{f«4t^-*iiuiQ.xe2itroid<]  s  0.0; 

for(Cl.xnt  s  1;  Cl^nt  <  »oum^entroid*;  01x01  +  +  ) 

{ 

caIc J'xov4r(m«trix-p,  fe4t..p,  Clxnt); 
print  JSiaiyxov4ri4nce(m4trtx^,  fe4t-p); 

perforin^-^PuasJlV[fe4t^— »nttmxentroids|  +s  «<irt(m4trix4>>-»P^et); 
perforin4)-*Sl  s  0.0; 

c4lcx«ntroidJProb(m4trix.p,  fcAt^t,  Clxnt  ); 

C4lcuiUt-PMLE(m4trix.^,  fext^,  Clxnt,  perform^)); 
perform^-^S2  +s  perform^t-^Sl; 

perform^-*Av.^enaity[fe4t4>-*numx«ntroid«]  +s  (perform4>-»Sl/aqrt(m4trix.p-»Fxiet)); 

} 

perform.4>*>«Av^enaity[fe4t.4>’-*nuraxentroid«]  =  perform^— »  A  v^ensity[f«4t +>>^000)  xentroida]/fe4t4>--*numxentroids; 
perform.^— »P4rtitiooJ3en«ity[fe4t^»*numx«ntroid«]  =  (perforiQ^*^S2/perform^-»Puzz  JiV{(e4t^-*ttumxentroid«]); 
if(PRINT^IAG)  printf(**Hypervoluin«  =  S2  =  9(«“n*',  perforro4><^Fu2z JIV[fe4t^>-»Dumxentroid6],  perform^— »S2  ); 

} 

void  »tore^crform4nce^4t4(CJPARAMS  aconfig^,  PJ^ARAMS  •perform4>) 

{ 

/*  ThU  function  output  the  Performxrce  reaults  for  4II  the 
numbers  of  clusters  evxluxted 
Nexle  Prescott  OCT  93 

V 

int  dust; 

printf(’*"n#  Cluster**tPu2ay  Hypervol“lAv  Density‘*tP4rtition  Density^n”); 
printf(  . “t . “t . “t . "n”); 

for(dust  s  config-p«*initi4lxlusters;  dust  <  config4>*^fin4lxlusters;  clust++) 

printf("  %d“t“t%e“t96e“t%e“n”,  dust,  perform-p-*Fu2zJHV(dust),  perform+^-^Av  JDensityfclust],  pcrform4>^P4rtition JDensity[dust] 
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B.2.4  Fuzzy  Classifier  -  FVQ.c. 


/• 


Progr4m  N»ine  :  FVQ.c 

Detcriptioa  :  Thi*  U  »  inodificAtioii  of  PuazCl-c  to  perform 
M  4  Pussy  Vector  Quantiser.  06  NOV  93. 

Author  :  Neale  Prescott 

University  :  USAP  Institute  of  Technology 

Date  :  08  NOV  93 

Other  Code  :  Numerical  Recipes  in  C  (2nd  £d) 

PussLIB.c 

PussLIB.h 

Input  Piles  :  datable,  setupJile 

References  :  1.  Unsupervised  optimal  Pussy  Clustering  by 

I. Gath  and  B.Geva,  IEEE  Transactions  on  Pattern 
Analysis  and  Machine  Intelligence,  Vol  11,  No. 7, 

July  1989. 

2.  A  Possibilistic  Approach  to  Clustering  by 
Krishnappuram  and  Keller,  IEEE  Transactions  on  Pussy 
Systems,  Vol  1,  No. 2,  May  1993. 

3.  Speech  Coding  Method  Using  Fussy  Vector  Quantisation 
by  ASAKAWA.  ICHIKAWA,  YAJIMA  and  YAMASAKI 
IEEE  ICASSP  1989,  pp755  •pp7fi8 

4.  A  2.4  KBPS  Speech  Coding  Method  Based  on  Pussy  Vector 
Quantisation  by  ASAKAWA,  YAMASAKI  and  ICHIKAWA 
IEEE  ICASSP  1990,  pp673-676 


#include  <stdio.h> 

#include  <stdlib.h> 

#include  <string.h> 

#include  <math.h> 

#include  <errno.h> 

#include  '’PussLIB.h” 
fjlinclude  ’’nrutil.h” 

#ifdef  DIAG 

#define  PRINT JDIAG  1 

#else 

#define  PRINT  J3IAG  0 
#endif 

void  FMLE_vector-quaRtise(C JPARAMS  •config.p,  M-PARAMS  *  .latrix-p,  F-PARAMS  efeat-j), 
PJ* ARAMS  eperform^,  double  eecov^,  double  edet^,  double  •aprtor.p); 

int  status; 

C^ARAMS  config.4>arams; 

FJ*ARAMS  feat^arams; 

MJPARAMS  mtx..params; 

SJ’ARAMS  stats^arams; 

P-PARAMS  perform.4>arams; 

double  ••COVuntx,  eDETjuts,  ePEXORIuntx; 

FILE  eclusterJn; 

main(int  argc,  char  eargvQ) 
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r6atd^onflgurfttion^lc(iecoaflg^«r«m*,  irfeat^tarami,  4rge,  4rgv); 

m4kejn«trices(&conflg4>4r4m*,  ^intx-p4r4m«,  4£«t4U^4r4m«,  ^perform-paraini); 

COV^tx  s  dm4trix(l,  (config4>4r4in«.fe4lure«^>er jvector«config^xr4m*.fiB4l^lubler6 )  , 

1,  config-p4r4ms.fe4ture«4>er. vector  J; 

PRIORl^tx  3  <lvector(l,  config4>4r4ms.fin4l^lu«ter«  ); 

DBTuntx  s  dvector(l,  config4)4r4m«.fin4l^lu«ter«  ); 

lo4d  Je4ture*(i:config4>4r4m«,  &mtx^4r4m«)i 

lo4d^entroidi(leconfig^>4r4nu,  Jemtx.p4r4m«,  COV^tx,  DET^tx,  PR10Rl_mtx); 

FMLE.vectorjqu4atUe(4econfig^4r4mft,  &mtK.p4r4m«,  &fe4t-p4r4m«,  j^perform^xrxme, 
COVjntx,  DETjntx.  PRIORI^tx); 

delete jn4trice«(&config^>4r4ms,  &iiitx-p4r4in«,  je«t4ta^4r4m<,  leperform4>4r4in«)i 

free^m4trix(COV^tx,  1,  (config^4r4mii.fe4ture«4>er-vector«coQflg^4r4m».fin4i jclu^ters)  , 
1,  config^4r4m«.fe4ture<^er.vector  ); 

free^vector(PRIORljntx,  1,  config^4r4m«.fin4l^lu«ter«  ); 

free^vector(DBT^tx,  1,  config^xrxme.fiaxl^lueter*  ); 

return(O): 

} 


void  FMLE.vector-qu4atUe(C J^ARAMS  econfig^,  MJ^ARAMS  em4trix-p,  FJ’ARAMS  efeat^, 
PJ*ARAMS  eperform^t,  double  eecov^,  double  edet^,  double  »4prior-p) 

{ 

iat  eittiter,  i,  j,  k,  best; 

double  tmp-ulist,  diet,  mexaulist,  V4rwdis(,  (ot^emb,  m4x^etnb>  meenunemb; 


k  =  1; 

for(cluster  si;  cluster  <  <e4t4>-~*numxentroids;  cluster4>4’) 

{ 

for(i  3  1;  i  <  config^-^feetures^er.vector;  !+•(•,  k+<f ) 
for(j  3  1;  j  <  config^'>*fe4ture6^}er_vector;  i*4'4’) 
ra4trix^>*F^ov4r Jzitx[i][}]  s  cov^[k][j]; 

metrix^-^F^et  3  det-p[cluster]; 
ro4trix4>’-»4PRI0RI  3  4prior4)[clustei]; 

cxlc^ist  J'MLE(m4trix4),  (e4t^,  cluster,  perform^); 

} 

computejnember-FKM(config^,  matrix^),  fe4t-p); 

dist  3  0.0; 

V4r.jdist  3  0.0; 
tot-memb  3  0.0; 

for(i  3  1;  i  <  config4>->*number.vectors;  1+  +  ) 

{ 

mxxunemb  3  0.0; 

for(j  3  1;  j  <  config..p— *fin4l.xlusters;  j  +  +  ) 
if(m4xjnemb  <  m4trix-p— •mshipsjntx[i][j]) 

{ 

m4X-memb  3  in4trix4>-^niships^itx[i][i]; 
best  3  j; 

) 

tot-memb  +=  mex-memb; 

tmp-dist  s  (log(m4trix-p— »dist-mtx[i]{best])); 
varuiist  +3  DSQR(tmpjdist); 
dist  tmp^ist; 


meenunemb  s  tot-memb  /  config.p-»nuD)ber-vectors  ; 
me4n.jdist  3  dist  /  config4>Mnumber.vectors; 

verwdist  3  (v4r.dist/a>nfig-p— •number-vectors)  —  DSQR(me40jdist); 
V4r.dist  3  sqrt(v4r^ist); 
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prinU(”Dutortion  (H«I  for  [HiJ  "t: 

config^>-»cltt«icrjiain«. 

di*t, 

(mooBjdUt  4  3«v*rutUl)» 
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B.2.5  makefile. 


t  FLTLT  D.I.Praacott 

•  Tliaala  Fuz;  Clnataring 

•  08  OCT  93 

•  FILE:  Bakaflla  for  Fuzzy  Coda  for  THESIS 

•  PUBPOSB:  Thia  flla  aill  alloa  antOBatlag  bnildiag  of  tha 

•  azacatabla  prograaa  daflnad. 

•  Tba  baat  rafaranca  for  HAEE  la  "Hanaglng  Projacta  aitb  aaka"  by  Oram  and  Talbot 
t  O’Eallly  t  iaaoclataa,  Inc. 


DEBUO.FLiO  ■  -g  8  alloaa  dabngging  to  aork 

lUTH.LIB  a  -la  t  location  of  C  iMtli  librarian 

OCC  a  gcc  8  location  of  ilSI-C  coa^llar  on  Baakaye 

DIP.FLAQ  a  -DDIAO  8  -DDIiG  Prlvata  dabngglng  flag  for  aarboaa  node 

8 

8 . 

F2  :F3.c  nr.h  FnzzLIB.o  nmtll.o  aoaant.o  dpythag.o  daadca^.o 

${0CC}  -o  F2  F2.C  FnzzLIB.o  nmtll.o  aoaant.o  dpythag.o  daadcap.o  ${DEBU6_FLA0}  ${H1TH_LIB}  ${DIP_FLAG} 
FVq  ;FTq.c  FnzzLIB.o  nmtll.o  aoaant.o  dpythag.o  daada^.o 

gcc  -o  FVQ  FVq.c  FnzzLIB.o  nmtll.o  aoaant.o  dpythag.o  daadcap.o  -la  $(cflaga)  -DSDB  -DDIIG 


fnkn  ;  fnkn.c  FnzzLIB.o  nmtll.o 

»{GCC}  -o  fnkn  fnkn.c  nmtll.o  ${BATH_LIB}  »<DIP.FLiG}  ${DKBUG.FLiG} 


8  aaziaally  optiaiaad  aaraion  of  F2  for  SUB 

8  Hota  ahola  coi^ilation  la  aithont  debng.  la  Fnzzlib.c  la  optiaiaad  too. 

• - 

F2opt  :F2.c  nr.h  FnzzLIB.c  nmtil.o  aoaant.o  dpythag.o  davdcap.o 

${GCC}  -o  F2opt  F2.C  FnzzLIB.c  nmtil.o  aoaant.o  dpythag.o  daTdcmp.o  ${IUTB_LIB}  -02 
FVQopt  :FVq.c  FnzzLIB.o  nmtil.o  aoaant.o  dpythag.o  dsvdcap.o 

gcc  -02  -o  FVQopt  FVQ.C  FnzzLIB.c  nmtil.o  aoaant.o  dpythag.o  dsvdcap.o  -la  -DSUH 


8  Library  rontinaa 

• - 

FnzzLIB.o  iFnzzLIB.c  FnzzLIB.h  nmtil.h  nr.h 
8<GCC>  -c  FnzzLIB.c  ${DEBUG.FUG}  *{DIP.FL1G> 


nmtil.o  :  nmtil.c 

${GCC>  -c  nmtU.c  8{DEBUG_FL1G} 

aoaant.o  :  aoaant.c  nmtil.h 
${GCC}  -c  aoaant.c  ${DEBUG_FL1G} 

dpythag.o  :  dpythag.c  nmtil.h 
${aCC}  -c  dpythag.c  ${DEBUG_FUa} 

davdcap.o  :  datdcap.c  nmtil.h 
8{GCC}  -c  dardc^.c  ${DEBUG_FUQ} 
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B.S  Simple  UFPONC  in  MATLAB  sci-ipt 


X  i  qsick  aaaity  ckack  of  FIM  and  UFPOIC 
X  laala  Proacot 

X - 

X  Olobal  coaatanta 

X - 

akBx_control<la  ■  4; 
apailon  ■  0.001; 

■az.ltar  ■  600; 


load  taat6.dat 
data  >taat6; 
claar  taatO 
Da.n]  >alaa(data}; 

Xforaat  abort  a 

X  Haka  tka  nacaaaary  aatrlcaa 

X - 

cantToida  ■  zaToa(Baz_cantroid8,  n) ; 
diatancaa  ■  zaroaCa,  ■az.cantrolda) ; 

■anbahlpa  >  zaroaCa,  aaz.centrolda) ; 
oldakipa  ■  zaroa(a,  aaz.cantroida) ; 

X  Inltlaliaa  tka  cantroids 

X - 

f irat.can  ■  aaan(data) ; 
cantroidad , ;  }afir8t_caii; 

cantroidaO, :)  ■  flrat.can  *  (firat.can  •  *  O.OOS) 

eantroidaO, :}  •  flrat.can  +  (firat.can  a  <-l)"2  a  0.006) 
claar  firat.can 

Xcantroidad , : )  a  data(5,:); 

Xcantroid8(2, ; )  •  datad?,;); 

X  Initialiaa  aoaa  rariablaa 

X - 

objactiva  a  lO; 

CLUST  a  3; 
track  a  [] ; 
itaratlona  a  O; 


X  Calcnlata  tka  diatancaa 

X - 

for  ial:a 
for  Jal: CLUST 

XanbV  a  data(i,;)  -  c8ntroid8( j , :} ; 
tq>  a  XsnbT  a  lanbV ’ ; 
diatancaad, j)  a  tap; 
and 

and 

X  Calcnlata  tka  aeabarakipa 

X - 

for  ial;B 

8na.all  a  o.O; 

FLAO  a  -1; 
for  jal; CLUST 

if(  diatancaad, j)  'a  o.O) 
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■  souall  >  1.0/dlataac«a(i,j)i 
FLAG  -  j; 


•nd 

for  j-l:CLOST 
ifCFLlO  —  -1) 

BOMbolklpsd, j)  ■  (1 .0/dlatanc«o(i, j))/oiBi_all: 

olaa 

if(J  “  FLIO) 

■aabahipad. j)  •  1.0; 
alao 

■oabalilpad, j)  ■  0.0; 
and 

and 

and 


and 


I - 

X  Main  oparationa  loop  -  Fnzzy  E-Maana 

I - 

Bhila  ((objactlaa  >  apallon)tdtaratlona  <  uz.itar)) 

for  i«l:CLUST 

nnaarator  •  zaroa(l,n); 
danoBinator  >0.0; 
for  j“l:B 

nnBorator  >  niuiarator  *  BOBbahipsCj,  i)*2  a  dataCj,:); 
danoBinator  <•  danoBinator  *  BaBbahip8(j,i)*2; 
and 

cantroidad, ;)  *  noBarator  ./  danoBinator; 
and 

oldahlpa  *  BaBbshipa; 

X  Calcnlata  tba  diatancaa 

X - 

for  :b 
for  jal:CL0ST 

XanbV  >  datad,:}  -  centroidaC  j  , :)  ; 
tap  a  XanbV  a  XanbV’; 
diatancaad, j)  a  tBp; 
and 
and 

X  Calcnlata  tha  BaBbarahipa 

X - 

for  lal;B 

anB_all  a  o.O; 

FLIO  a  -1; 
for  Jal:CL0ST 

if(  diatancaad, j)  'a  o.O) 

anB.all  a  80B_all  *  1 .O/diatancaaCi, j) ; 
alaa 

FLAG  a  j; 
and 

and 

for  jal:CLUST 
IfCFLAG  a»  -1) 

BOBbahipad, j)  a  (1 .0/diatancaad,J))/8nB_all; 

olaa 

if(j  a-  flag) 
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■•■bslklpad.J)  ~  1.0; 

•la* 

M^shipadij)  •>  0.0; 
•sd 

•ad 

•nd 

•nd 


X  Calcvlat*  th*  object it*  fonction 

X - 

objective  •  auu((Bui(aba(oldahips  -  BeBbships))) ; 
track  •  [track;  objective]; 
iteration*  ■  iterations  *  1 


•nd 

plot (track) 

titleC’FIH  -  Objective  function’) 

xlabeK’Iterat  ions  ’ ) 

grid 


X  Initialise  the  Fnaxy  covariance  natriz 

X - 

fnzz.cov  eeje(n); 

X  Calculate  the  distances  (exponential) 

X - 

for  1>1:b 

for  j-l:CL0ST 
P-0.0; 
for  i-l:B 

P  -  P  +  BeBbships (i,j)/B; 

•nd 

IsnbV  -  datad,:)  --  centroids(j , : )  ; 
dlstancesd, j)  »  (1 .0/P) *ezp( (IsnbV  •  IsubV»)/2); 
•nd 

•nd 

X  Calcnlate  the  BeBberships 


for  i-l:B 

snB_all  -  0.0; 

FL16  -  -1; 
for  j-l:CLUST 

lf(  diatancesd, j)  0.0) 

snB_all  -  soBi_all  *  1 .0/diatances(i, j) ; 
else 

FLiO  -  j; 

•nd 

•nd 

for  j-l:CLDST 
lf(FU0  »  -1) 

BeBbships (i,j)  ■  (1 .0/di8tances(i, j))/snB_all; 

else 

if(j  —  FLiO) 

BeBbships(i,j)  -  1.0; 

else 

BOBbshipedij)  -  0.0; 

•nd 

•nd 

•nd 

•nd 
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lt«r>tlona  ■  0; 
objaeti**  •  10; 
txackSaQ ; 

X - 

X  Rain  oparationa  laap  -  Fuzzy  Haziaum  Likalibaad  Eatiaatlan 

X - 

ahlla  ((abjactiza  >  apsllan)k(itaratiaiLS  <  aaz.itar)) 

X  Calcnlata  tha  caotraids 
X - 

iar  i«l;CL0ST 

aaaazataz  ■  zazasvl.n); 
danaalnatar  >0.0; 
far  j“l:a 

anaaratar  ■  amiazatar  +  BaBbabips(j,i)~2  •  dataCj,;); 
danaainatar  •  danaainatar  +  aaabships(j,l)~2; 
and 

centraldsd, :)  >  noaarater  ./  danaainatar; 

and 

X  Calcnlata  tha  Fnzzj  cazarianca  aatriz  far  tha  cantraida 

X - 

far  j»l:CL0ST 
P-0.0; 
far  1-1 :n 

P  *  P  ♦  aaabahipaCi, j)/a; 
and 

danaa  ■  0.0; 
far  1-1: a 

XznbV  ■  dataCl,:)  -  eantraida( j , ; 

fnzz.caz  -  fnzz.caz  +  aaabahlpsd, j)  .a  (XanbV’  a  isnbV); 
danaa  -  danaa  *  aaabshlpad, j) ; 
and 

fnzz.caz  -  fnzz.caz  ./  danaa; 
sqrt.F.dat  -  sqrt (dat (fnzz.caz) ) ; 
far  1-1: a 

ZsnbV  -  datad,  :)-cantreids(j, :) ; 

diatancasd, j)  -  (8qrt.F.dat/P)aazp((l8nbV  a  InzCfnzz.cez)  a  X8nbV')/2) 
and 
and 

aldshlps  -  aaabahipa; 

X  Calcnlata  tha  aaid>ar8hip8 

X - 

far  1-1 :a 

8na.all  -  0.0; 

FL*8  -  -1; 
far  j-l:CLUST 

lf(  diatancasd, J)  '-  0.0} 

8na.all  -  ana_all  a  1 .0/distancasd,  j) ; 
alsa 

FLIO  -  j; 

and 

and 

far  j-l:CL0ST 
IfCFLiQ  —  -1) 

aaabshipBdiJ)  -  (1 .0/di8tancaad,j))/8na.all; 

alsa 

if(j  “  FLIO) 

aaabBhipsd, j)  -  1.0; 

alsa 
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*  • 


■aabshlpsd.J)  ■  0.0; 
•lul 

•ad 

•ad 


•ad 

t  Calcalata  tha  objacti*a  fanctioa 

X - 

objactlva  •  Baz(Baz(aba(oldsbips  -  ■•■bahlpa))) ; 
tracU  •  [track2;  objactiva]  ; 

Itaratloaa  ■  itaratloaa  *  1 


•ad 

flgura 

plot(track2) 

titla(*FflLE  -  Objactiva  faactioa*) 

zlabal ( > Itarat loaa  * ) 

grid 
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B.4  Code  to  Determine  the  Number  of  Hidden  Nodes 

% 

t  Hatlab  Script  file  -  D.l. Prescott 


X  Cslcnlatlon  of  Hlddoa  lodaa  for  a  lonral  lot 
X 

X  Bomia’a  Foranila  (Vidros) 

X 

X  10  *(  [loK.ftrs  '•'l]«|odoa  +  (lodas*l)*ClnB_oatpats]  )  <  ■iiB_data_Ba^las 
X 

X  og. 

X 

X  20  diBoaslonal  apaoch  faatnraa 
X  12  apaakara 

X  20,000  aaapla  faatara  sactora 
X 

X  10(  [2(Hl]a|  +  (m)al2  )  <  20,000 
X 

X  »  33aa  a  12  <  2000 
X 

X  ->  ■  <  (2000-12) /33 

X 

X  ->  I  <  60.24 


diaa  >  20; 
nna.aaiq>laa  ■  1000; 
nim_claaaaa  >12; 
loops  a  20; 

iltatr  B  zaro8(loopsai,2); 

IVlBtzd.S)  bO; 

for  iBl; loops 

■odas  B  ((naa.saBple8/10)-iniB_cla8saa)/(dias+lanaa_clas8e8) ; 
Ubtz(ial,l)  B  noa_8aBpla8; 

UlBtz(ial,2)  B  lodaB; 
iiaB_8aBplas  •  onBi_aai7la8  a  1000; 
and 

8tair8(Uhtx( :  ,2)) 

tltla( ’Raziaoa  rnaabar  of  hidden  nodaa  for  data  aa^laa,  12  claasea,  20diaa’) 
zlabaK’SsBplea  X  1000’) 
ylabaK  ’Hidden  nodaa  > ) 
grid 
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B.5  FKM  configuration  file 


Listed  below  is  a  typical  configuration  file  for  the  UFP-ONC  program.  The  leading 
numbers  are  not  included  they  are  for  explanation  only.  This  configuration  file  is  for  the 
Anderson  iris  data  set. 

1.  %  Test  configuration  file  FKM,  UFP-ONC,  Neale  Prescott  150CT  93 

2.  iris.dat 

3.  X 

4.  iris. dust 

5.  iris.perfm 

6.  U 

7.  150 

8.  4 

9.  2 

10.  6 

11.  2 

12.  0.001 

13.  1000 

1.  Comment  line 

2.  Feature  vector  (data  file)  name 

3.  The  cluster  file  name 

4.  The  performance  file  name  'NOT  USED  but  REQUIRED  FOR  STARTUP) 

5.  The  mode  of  operation 

•  U  for  UFP-ONC,  ie  both  FKM  and  FMLE 

•  F  for  FKM  only 

6.  Number  of  feature  vectors 

7.  Dimension  of  feature  vectors 

8.  Number  of  initial  clusters 

9.  Final  number  of  clusters 

10.  Fuzziness,  generally  use  2 

11.  Epsilon,  the  stopping  criteria,  must  be  [0, 1] 

12.  Maximum  number  of  iterations  before  stopping,  ie  If  not  converged  in  1000  iterations 
then  stop. 
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B.6  ANN  configuration  file 


datafile: 
hiddenlo : 
hiddenhi : 
hiddenint : 
eta: 
max err : 
m2ucepochs : 


timitday02.a8C 

20 

20 

20 

0.30 

0.01 

10000 
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